[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:43:05 dist-test-slave-j5c3 krb5kdc[1938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:43:05 dist-test-slave-j5c3 krb5kdc[1938](info): set up 2 sockets
Sep 28 04:43:05 dist-test-slave-j5c3 krb5kdc[1938](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 04:43:06.737138 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:43:06 dist-test-slave-j5c3 krb5kdc[1938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:43:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:43:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:43:06.859032 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:46841
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:34303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:46841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:06.944051 1951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:06.944413 1951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:06.944445 1951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:06.945930 1951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:43:06.945990 1951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:06.946009 1951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:06.946025 1951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:43:06.946041 1951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:43:06.947842 1951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:34303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:46841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:46841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:06.948196 1951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:06.948523 1951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:06.955852 1959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:06.955871 1957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:06.955852 1956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:06.956193 1951 server_base.cc:1034] running on GCE node
I20260928 04:43:06.956429 1951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:06.956710 1951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:06.957893 1951 hybrid_clock.cc:648] HybridClock initialized: now 1790570586957881 us; error 40 us; skew 500 ppm
Sep 28 04:43:06 dist-test-slave-j5c3 krb5kdc[1938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570586, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:06.960500 1951 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:43:06.960978 1951 webserver.cc:460] Webserver started at http://127.1.226.254:36909/ using document root <none> and password file <none>
I20260928 04:43:06.961347 1951 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:06.961400 1951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:06.961575 1951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:06.963079 1951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "9fca96f4630b48c1bea75e5ce6bd2498"
format_stamp: "Formatted at 2026-09-28 04:43:06 on dist-test-slave-j5c3"
server_key: "4eeba74056bc657e298f7a3f30ea8614"
server_key_iv: "a2f1b9c7bbdbdf908e654eb7745bdfd8"
server_key_version: "encryptionkey@0"
I20260928 04:43:06.963747 1951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "9fca96f4630b48c1bea75e5ce6bd2498"
format_stamp: "Formatted at 2026-09-28 04:43:06 on dist-test-slave-j5c3"
server_key: "4eeba74056bc657e298f7a3f30ea8614"
server_key_iv: "a2f1b9c7bbdbdf908e654eb7745bdfd8"
server_key_version: "encryptionkey@0"
I20260928 04:43:06.965443 1951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260928 04:43:06.966236 1966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:06.966522 1951 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:06.966652 1951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "9fca96f4630b48c1bea75e5ce6bd2498"
format_stamp: "Formatted at 2026-09-28 04:43:06 on dist-test-slave-j5c3"
server_key: "4eeba74056bc657e298f7a3f30ea8614"
server_key_iv: "a2f1b9c7bbdbdf908e654eb7745bdfd8"
server_key_version: "encryptionkey@0"
I20260928 04:43:06.966741 1951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:06.980357 1951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:06.982535 1951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:06.982661 1951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:06.986699 1951 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:46841
I20260928 04:43:06.986740 2018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:46841 every 8 connection(s)
I20260928 04:43:06.987139 1951 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:43:06.988098 2019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:06.990530 2019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fca96f4630b48c1bea75e5ce6bd2498: Bootstrap starting.
I20260928 04:43:06.991077 2019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fca96f4630b48c1bea75e5ce6bd2498: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:06.991323 2019 log.cc:824] T 00000000000000000000000000000000 P 9fca96f4630b48c1bea75e5ce6bd2498: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:06.992022 2019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fca96f4630b48c1bea75e5ce6bd2498: No bootstrap required, opened a new log
I20260928 04:43:06.993671 2019 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fca96f4630b48c1bea75e5ce6bd2498 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fca96f4630b48c1bea75e5ce6bd2498" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 46841 } }
I20260928 04:43:06.993824 2019 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fca96f4630b48c1bea75e5ce6bd2498 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:06.993848 2019 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fca96f4630b48c1bea75e5ce6bd2498 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fca96f4630b48c1bea75e5ce6bd2498, State: Initialized, Role: FOLLOWER
I20260928 04:43:06.993983 2019 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fca96f4630b48c1bea75e5ce6bd2498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fca96f4630b48c1bea75e5ce6bd2498" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 46841 } }
I20260928 04:43:06.994059 2019 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fca96f4630b48c1bea75e5ce6bd2498 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:43:06.994081 2019 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fca96f4630b48c1bea75e5ce6bd2498 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:43:06.994110 2019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fca96f4630b48c1bea75e5ce6bd2498 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:06.994642 2019 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fca96f4630b48c1bea75e5ce6bd2498 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fca96f4630b48c1bea75e5ce6bd2498" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 46841 } }
I20260928 04:43:06.994727 2019 leader_election.cc:302] T 00000000000000000000000000000000 P 9fca96f4630b48c1bea75e5ce6bd2498 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fca96f4630b48c1bea75e5ce6bd2498; no voters:
I20260928 04:43:06.994890 2019 leader_election.cc:288] T 00000000000000000000000000000000 P 9fca96f4630b48c1bea75e5ce6bd2498 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:43:06.994915 2022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fca96f4630b48c1bea75e5ce6bd2498 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:06.995019 2022 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fca96f4630b48c1bea75e5ce6bd2498 [term 1 LEADER]: Becoming Leader. State: Replica: 9fca96f4630b48c1bea75e5ce6bd2498, State: Running, Role: LEADER
I20260928 04:43:06.995121 2022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fca96f4630b48c1bea75e5ce6bd2498 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fca96f4630b48c1bea75e5ce6bd2498" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 46841 } }
I20260928 04:43:06.995164 2019 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fca96f4630b48c1bea75e5ce6bd2498 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:43:06.995707 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 1951
I20260928 04:43:06.995844 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:43:06.996724 1931 external_mini_cluster.cc:1428] Setting key 64c18d6a7c964f5403a550151ac0ac3e
I20260928 04:43:06.996994 2024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fca96f4630b48c1bea75e5ce6bd2498 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fca96f4630b48c1bea75e5ce6bd2498" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fca96f4630b48c1bea75e5ce6bd2498" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 46841 } } }
I20260928 04:43:06.997104 2024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fca96f4630b48c1bea75e5ce6bd2498 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:06.997339 2023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fca96f4630b48c1bea75e5ce6bd2498 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fca96f4630b48c1bea75e5ce6bd2498. Latest consensus state: current_term: 1 leader_uuid: "9fca96f4630b48c1bea75e5ce6bd2498" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fca96f4630b48c1bea75e5ce6bd2498" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 46841 } } }
I20260928 04:43:06.997568 2023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fca96f4630b48c1bea75e5ce6bd2498 [sys.catalog]: This master's current role is: LEADER
W20260928 04:43:06.997776 2036 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9fca96f4630b48c1bea75e5ce6bd2498: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:43:06.997879 2036 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:43:06.997989 2037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:43:06.998167 2037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:43:06.999857 2037 catalog_manager.cc:1347] Generated new cluster ID: 106303762c034023a3223b9e60a11999
I20260928 04:43:06.999917 2037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:43:07.008922 2037 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:43:07.009687 2037 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 04:43:07 dist-test-slave-j5c3 krb5kdc[1938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:07.012862 2037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fca96f4630b48c1bea75e5ce6bd2498: Generated new TSK 0
I20260928 04:43:07.013051 2037 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:43:07.017107 2040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:06.997841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46140 (local address 127.1.226.254:46841)
0928 04:43:07.000270 (+ 2429us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:07.000281 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:07.000322 (+ 41us) server_negotiation.cc:407] Connection header received
0928 04:43:07.000643 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:07.000658 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:07.000717 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:07.000867 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:07.003122 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:07.004918 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:07.006687 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:07.007480 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:07.007619 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:07.012360 (+ 4741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:07.012366 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:07.012366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:07.012384 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:07.014614 (+ 2230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:07.015285 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:07.015289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:07.015289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:07.015436 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:07.015830 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:07.015832 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:07.015832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:07.016215 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:07.016377 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:07.016677 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 04:43:07.016853 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2381,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.1.226.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
I20260928 04:43:07.043550 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:0
--local_ip_for_outbound_sockets=127.1.226.193
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:46841
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:34303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:07.126389 2045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:07.126601 2045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:07.126626 2045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:07.127882 2045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:07.127924 2045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:07.127952 2045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:43:07.129460 2045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:34303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:46841
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:07.129745 2045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:07.129977 2045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:07.135228 2050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:07.135360 2053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:07.135406 2051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:07.135635 2045 server_base.cc:1034] running on GCE node
I20260928 04:43:07.135834 2045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:07.136055 2045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:07.137225 2045 hybrid_clock.cc:648] HybridClock initialized: now 1790570587137205 us; error 30 us; skew 500 ppm
Sep 28 04:43:07 dist-test-slave-j5c3 krb5kdc[1938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570587, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:07.139875 2045 init.cc:377] Logged in from keytab as kudu/127.1.226.193@KRBTEST.COM (short username kudu)
I20260928 04:43:07.140367 2045 webserver.cc:460] Webserver started at http://127.1.226.193:43217/ using document root <none> and password file <none>
I20260928 04:43:07.140584 2045 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:07.140628 2045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:07.140802 2045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:07.141891 2045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-0/data/instance:
uuid: "54e83e02e54f41c8bd5770b9e1b21812"
format_stamp: "Formatted at 2026-09-28 04:43:07 on dist-test-slave-j5c3"
server_key: "c2ad3edf3f1c6739c25e81526869aaca"
server_key_iv: "35b55595f1b381a2e3e8dd7d19103a58"
server_key_version: "encryptionkey@0"
I20260928 04:43:07.142256 2045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance:
uuid: "54e83e02e54f41c8bd5770b9e1b21812"
format_stamp: "Formatted at 2026-09-28 04:43:07 on dist-test-slave-j5c3"
server_key: "c2ad3edf3f1c6739c25e81526869aaca"
server_key_iv: "35b55595f1b381a2e3e8dd7d19103a58"
server_key_version: "encryptionkey@0"
I20260928 04:43:07.143563 2045 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:43:07.144337 2060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:07.144500 2045 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:43:07.144588 2045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "54e83e02e54f41c8bd5770b9e1b21812"
format_stamp: "Formatted at 2026-09-28 04:43:07 on dist-test-slave-j5c3"
server_key: "c2ad3edf3f1c6739c25e81526869aaca"
server_key_iv: "35b55595f1b381a2e3e8dd7d19103a58"
server_key_version: "encryptionkey@0"
I20260928 04:43:07.144656 2045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:07.154318 2045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:07.156628 2045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:07.156792 2045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:07.157034 2045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:07.157338 2045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:07.157371 2045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:07.157394 2045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:07.157434 2045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:07.163081 2045 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:40681
I20260928 04:43:07.163132 2173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:40681 every 8 connection(s)
I20260928 04:43:07.163461 2045 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
I20260928 04:43:07.168365 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 2045
I20260928 04:43:07.168452 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance
I20260928 04:43:07.168607 1931 external_mini_cluster.cc:1428] Setting key e88714f515364d13e874ab78424380e0
Sep 28 04:43:07 dist-test-slave-j5c3 krb5kdc[1938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570587, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:07.173633 2040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:07.164266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:52229 (local address 127.1.226.254:46841)
0928 04:43:07.164343 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:07.164346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:07.164529 (+ 183us) server_negotiation.cc:407] Connection header received
0928 04:43:07.165072 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:07.165073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:07.165084 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:07.165135 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:07.166243 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:07.166741 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:07.167814 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:07.168003 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:07.168054 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:07.170696 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:07.170698 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:07.170698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:07.170706 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:07.172230 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:07.172650 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:07.172651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:07.172652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:07.172714 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:07.172972 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:07.172973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:07.172974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:07.173095 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:07.173168 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:07.173443 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 04:43:07.173540 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 04:43:07.173822 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:07.164223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:46841 (local address 127.1.226.193:52229)
0928 04:43:07.164402 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:07.164441 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:07.164981 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:07.165204 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:07.165207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:07.165947 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:07.166152 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:07.166158 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:07.166849 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:07.166851 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:07.167723 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:07.167726 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:07.168084 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:07.168085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:07.168292 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:07.168928 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:07.168937 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:07.170574 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:07.172323 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:07.172331 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:07.172334 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:07.172560 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:07.172783 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:07.172784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:07.172785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:07.172906 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:07.173159 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:07.173166 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:07.173313 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 04:43:07.173421 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 04:43:07.173581 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":106,"threads_started":1}
I20260928 04:43:07.174337 2174 heartbeater.cc:347] Connected to a master server at 127.1.226.254:46841
I20260928 04:43:07.174456 2174 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:07.174656 2174 heartbeater.cc:511] Master 127.1.226.254:46841 requested a full tablet report, sending...
I20260928 04:43:07.175288 1983 ts_manager.cc:194] Registered new tserver with Master: 54e83e02e54f41c8bd5770b9e1b21812 (127.1.226.193:40681)
I20260928 04:43:07.176352 1983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.193@KRBTEST.COM'} at 127.1.226.193:52229
WARNING: no policy specified for kudu/127.1.226.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
I20260928 04:43:07.194903 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:0
--local_ip_for_outbound_sockets=127.1.226.194
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:46841
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:34303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:07.280663 2179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:07.280864 2179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:07.280886 2179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:07.281972 2179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:07.282008 2179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:07.282034 2179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:43:07.283409 2179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:34303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:46841
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:07.283607 2179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:07.283774 2179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:07.288712 2184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:07.288756 2179 server_base.cc:1034] running on GCE node
W20260928 04:43:07.288724 2185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:07.288718 2187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:07.289106 2179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:07.289369 2179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:07.290514 2179 hybrid_clock.cc:648] HybridClock initialized: now 1790570587290486 us; error 45 us; skew 500 ppm
Sep 28 04:43:07 dist-test-slave-j5c3 krb5kdc[1938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570587, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:07.293438 2179 init.cc:377] Logged in from keytab as kudu/127.1.226.194@KRBTEST.COM (short username kudu)
I20260928 04:43:07.293918 2179 webserver.cc:460] Webserver started at http://127.1.226.194:35509/ using document root <none> and password file <none>
I20260928 04:43:07.294157 2179 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:07.294237 2179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:07.294378 2179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:07.295491 2179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-1/data/instance:
uuid: "5a2166f5423a4f9684ba371e6f866c96"
format_stamp: "Formatted at 2026-09-28 04:43:07 on dist-test-slave-j5c3"
server_key: "6d011ec004b4365c9fe5a2b54cd3f348"
server_key_iv: "94e49010a2027553efd6c217949277da"
server_key_version: "encryptionkey@0"
I20260928 04:43:07.295929 2179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance:
uuid: "5a2166f5423a4f9684ba371e6f866c96"
format_stamp: "Formatted at 2026-09-28 04:43:07 on dist-test-slave-j5c3"
server_key: "6d011ec004b4365c9fe5a2b54cd3f348"
server_key_iv: "94e49010a2027553efd6c217949277da"
server_key_version: "encryptionkey@0"
I20260928 04:43:07.297515 2179 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:43:07.298269 2194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:07.298444 2179 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:43:07.298542 2179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "5a2166f5423a4f9684ba371e6f866c96"
format_stamp: "Formatted at 2026-09-28 04:43:07 on dist-test-slave-j5c3"
server_key: "6d011ec004b4365c9fe5a2b54cd3f348"
server_key_iv: "94e49010a2027553efd6c217949277da"
server_key_version: "encryptionkey@0"
I20260928 04:43:07.298609 2179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:07.315817 2179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:07.317910 2179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:07.318066 2179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:07.318303 2179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:07.318650 2179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:07.318686 2179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:07.318708 2179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:07.318722 2179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:07.325215 2179 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:41015
I20260928 04:43:07.325255 2307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:41015 every 8 connection(s)
I20260928 04:43:07.325582 2179 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
I20260928 04:43:07.330273 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 2179
I20260928 04:43:07.330390 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance
I20260928 04:43:07.330564 1931 external_mini_cluster.cc:1428] Setting key 472b34ea2e9e1c76b5cf889f66f9d962
Sep 28 04:43:07 dist-test-slave-j5c3 krb5kdc[1938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570587, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:07.336221 2040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:07.326703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:45939 (local address 127.1.226.254:46841)
0928 04:43:07.326765 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:07.326766 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:07.326984 (+ 218us) server_negotiation.cc:407] Connection header received
0928 04:43:07.327524 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:07.327525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:07.327538 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:07.327591 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:07.328601 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:07.329179 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:07.330370 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:07.330666 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:07.330764 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:07.333257 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:07.333259 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:07.333259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:07.333268 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:07.334850 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:07.335241 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:07.335242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:07.335242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:07.335306 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:07.335542 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:07.335543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:07.335543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:07.335674 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:07.335746 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:07.336014 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 04:43:07.336094 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 04:43:07.336436 2310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:07.326682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:46841 (local address 127.1.226.194:45939)
0928 04:43:07.326855 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:07.326898 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:07.327418 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:07.327677 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:07.327680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:07.328294 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:07.328502 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:07.328506 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:07.329279 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:07.329280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:07.330247 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:07.330249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:07.330767 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:07.330768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:07.330917 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:07.331555 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:07.331564 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:07.333159 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:07.334920 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:07.334927 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:07.334929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:07.335156 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:07.335366 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:07.335367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:07.335367 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:07.335475 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:07.335737 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:07.335744 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:07.335886 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 04:43:07.335993 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 04:43:07.336181 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260928 04:43:07.336869 2308 heartbeater.cc:347] Connected to a master server at 127.1.226.254:46841
I20260928 04:43:07.336951 2308 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:07.337136 2308 heartbeater.cc:511] Master 127.1.226.254:46841 requested a full tablet report, sending...
I20260928 04:43:07.337539 1983 ts_manager.cc:194] Registered new tserver with Master: 5a2166f5423a4f9684ba371e6f866c96 (127.1.226.194:41015)
I20260928 04:43:07.337914 1983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.194@KRBTEST.COM'} at 127.1.226.194:45939
WARNING: no policy specified for kudu/127.1.226.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
I20260928 04:43:07.355445 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:0
--local_ip_for_outbound_sockets=127.1.226.195
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:46841
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:34303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:07.439373 2313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:07.439631 2313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:07.439658 2313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:07.440905 2313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:07.440953 2313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:07.440981 2313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:43:07.442461 2313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:34303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:46841
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:07.442727 2313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:07.442966 2313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:07.448136 2321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:07.448210 2319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:07.448233 2313 server_base.cc:1034] running on GCE node
W20260928 04:43:07.448177 2318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:07.448590 2313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:07.448901 2313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:07.450089 2313 hybrid_clock.cc:648] HybridClock initialized: now 1790570587450065 us; error 38 us; skew 500 ppm
Sep 28 04:43:07 dist-test-slave-j5c3 krb5kdc[1938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570587, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:07.452730 2313 init.cc:377] Logged in from keytab as kudu/127.1.226.195@KRBTEST.COM (short username kudu)
I20260928 04:43:07.453109 2313 webserver.cc:460] Webserver started at http://127.1.226.195:44803/ using document root <none> and password file <none>
I20260928 04:43:07.453298 2313 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:07.453336 2313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:07.453423 2313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:07.454416 2313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-2/data/instance:
uuid: "3abd43cf269e4931bf5ca19dbe5cff49"
format_stamp: "Formatted at 2026-09-28 04:43:07 on dist-test-slave-j5c3"
server_key: "94979222a96a6840957ddf7689d43a4f"
server_key_iv: "f3e377e219a539bfd66b66ec205f7c6e"
server_key_version: "encryptionkey@0"
I20260928 04:43:07.454725 2313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance:
uuid: "3abd43cf269e4931bf5ca19dbe5cff49"
format_stamp: "Formatted at 2026-09-28 04:43:07 on dist-test-slave-j5c3"
server_key: "94979222a96a6840957ddf7689d43a4f"
server_key_iv: "f3e377e219a539bfd66b66ec205f7c6e"
server_key_version: "encryptionkey@0"
I20260928 04:43:07.456009 2313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 04:43:07.456888 2328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:07.457100 2313 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:07.457224 2313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "3abd43cf269e4931bf5ca19dbe5cff49"
format_stamp: "Formatted at 2026-09-28 04:43:07 on dist-test-slave-j5c3"
server_key: "94979222a96a6840957ddf7689d43a4f"
server_key_iv: "f3e377e219a539bfd66b66ec205f7c6e"
server_key_version: "encryptionkey@0"
I20260928 04:43:07.457319 2313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:07.488243 2313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:07.490255 2313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:07.490370 2313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:07.490556 2313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:07.490871 2313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:07.490926 2313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:07.490969 2313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:07.491000 2313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:07.498027 2313 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:45587
I20260928 04:43:07.498070 2441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:45587 every 8 connection(s)
I20260928 04:43:07.498437 2313 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
I20260928 04:43:07.500357 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 2313
I20260928 04:43:07.500439 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance
I20260928 04:43:07.500701 1931 external_mini_cluster.cc:1428] Setting key bebdb8088340426abf57f55ca3fe1065
Sep 28 04:43:07 dist-test-slave-j5c3 krb5kdc[1938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570587, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:07.508704 2040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:07.499197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:56157 (local address 127.1.226.254:46841)
0928 04:43:07.499260 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:07.499262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:07.499919 (+ 657us) server_negotiation.cc:407] Connection header received
0928 04:43:07.500536 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:07.500537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:07.500550 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:07.500619 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:07.501794 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:07.502414 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:07.503416 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:07.503608 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:07.503675 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:07.505891 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:07.505893 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:07.505893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:07.505901 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:07.507396 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:07.507791 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:07.507792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:07.507792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:07.507848 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:07.508063 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:07.508063 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:07.508063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:07.508220 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:07.508277 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:07.508538 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 04:43:07.508604 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 04:43:07.508890 2444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:07.499203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:46841 (local address 127.1.226.195:56157)
0928 04:43:07.499347 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:07.499387 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:07.500388 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:07.500706 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:07.500710 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:07.501440 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:07.501700 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:07.501705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:07.502513 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:07.502514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:07.503308 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:07.503311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:07.503686 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:07.503687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:07.503788 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:07.504406 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:07.504414 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:07.505788 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:07.507493 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:07.507499 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:07.507500 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:07.507719 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:07.507907 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:07.507908 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:07.507908 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:07.508008 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:07.508282 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:07.508288 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:07.508417 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 04:43:07.508509 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 04:43:07.508640 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
I20260928 04:43:07.509415 2442 heartbeater.cc:347] Connected to a master server at 127.1.226.254:46841
I20260928 04:43:07.509492 2442 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:07.509682 2442 heartbeater.cc:511] Master 127.1.226.254:46841 requested a full tablet report, sending...
I20260928 04:43:07.510071 1983 ts_manager.cc:194] Registered new tserver with Master: 3abd43cf269e4931bf5ca19dbe5cff49 (127.1.226.195:45587)
I20260928 04:43:07.510491 1983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.195@KRBTEST.COM'} at 127.1.226.195:56157
I20260928 04:43:07.513012 1931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 04:43:07 dist-test-slave-j5c3 krb5kdc[1938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570587, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 04:43:07 dist-test-slave-j5c3 krb5kdc[1938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570587, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.226.193@KRBTEST.COM
I20260928 04:43:07.536690 2449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:07.529034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38494 (local address 127.1.226.193:40681)
0928 04:43:07.529195 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:07.529200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:07.529211 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:43:07.529238 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:07.529240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:07.529254 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:07.529349 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:07.529910 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:07.530620 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:07.531328 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:07.531640 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:07.531693 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:07.533946 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:07.533955 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:07.533956 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:07.533963 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:07.535532 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:07.535853 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:07.535855 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:07.535855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:07.535909 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:07.536079 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:07.536079 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:07.536080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:07.536281 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:07.536426 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:07.536492 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 04:43:07.536558 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
W20260928 04:43:07.537048 2088 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38494
Sep 28 04:43:07 dist-test-slave-j5c3 krb5kdc[1938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 04:43:07 dist-test-slave-j5c3 krb5kdc[1938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.193@KRBTEST.COM
I20260928 04:43:07.558991 2449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:07.552067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38504 (local address 127.1.226.193:40681)
0928 04:43:07.552142 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:07.552144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:07.552288 (+ 144us) server_negotiation.cc:407] Connection header received
0928 04:43:07.552340 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:07.552341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:07.552354 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:07.552407 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:07.552990 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:07.553478 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:07.554227 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:07.554442 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:07.554500 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:07.556822 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:07.556823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:07.556823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:07.556856 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:07.558073 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:07.558383 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:07.558384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:07.558384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:07.558435 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:07.558584 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:07.558585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:07.558585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:07.558674 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:07.558764 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:07.558853 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 04:43:07.558917 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 04:43:07.559593 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 2045
I20260928 04:43:07.564438 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 2179
I20260928 04:43:07.570303 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 2313
I20260928 04:43:07.574504 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 1951
2026-09-28T04:43:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1886 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:43:07 dist-test-slave-j5c3 krb5kdc[2456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:43:07 dist-test-slave-j5c3 krb5kdc[2456](info): set up 2 sockets
Sep 28 04:43:07 dist-test-slave-j5c3 krb5kdc[2456](info): commencing operation
krb5kdc: starting...
W20260928 04:43:08.619807 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:43:08 dist-test-slave-j5c3 krb5kdc[2456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:43:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:43:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:43:08.742796 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:40145
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:40145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:08.822894 2469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:08.823136 2469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:08.823172 2469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:08.824338 2469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:43:08.824404 2469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:08.824425 2469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:08.824446 2469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:43:08.824474 2469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:43:08.825974 2469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:40145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:40145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:08.826242 2469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:08.826467 2469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:08.831313 2474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:08.831305 2475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:08.831453 2469 server_base.cc:1034] running on GCE node
W20260928 04:43:08.831460 2477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:08.831851 2469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:08.832137 2469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:08.833333 2469 hybrid_clock.cc:648] HybridClock initialized: now 1790570588833321 us; error 43 us; skew 500 ppm
Sep 28 04:43:08 dist-test-slave-j5c3 krb5kdc[2456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570588, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:08.836110 2469 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:43:08.836615 2469 webserver.cc:460] Webserver started at http://127.1.226.254:44491/ using document root <none> and password file <none>
I20260928 04:43:08.836869 2469 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:08.836935 2469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:08.837078 2469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:08.838124 2469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "d033df4fa9f94e8f9a95c19fbb1795b0"
format_stamp: "Formatted at 2026-09-28 04:43:08 on dist-test-slave-j5c3"
server_key: "fb6ef411e7df3b64983132fcdfbb2ed1"
server_key_iv: "8b5b7d2705329cbf4f91727fb4cbf388"
server_key_version: "encryptionkey@0"
I20260928 04:43:08.838479 2469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "d033df4fa9f94e8f9a95c19fbb1795b0"
format_stamp: "Formatted at 2026-09-28 04:43:08 on dist-test-slave-j5c3"
server_key: "fb6ef411e7df3b64983132fcdfbb2ed1"
server_key_iv: "8b5b7d2705329cbf4f91727fb4cbf388"
server_key_version: "encryptionkey@0"
I20260928 04:43:08.839892 2469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:43:08.840739 2484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:08.840907 2469 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:43:08.841003 2469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "d033df4fa9f94e8f9a95c19fbb1795b0"
format_stamp: "Formatted at 2026-09-28 04:43:08 on dist-test-slave-j5c3"
server_key: "fb6ef411e7df3b64983132fcdfbb2ed1"
server_key_iv: "8b5b7d2705329cbf4f91727fb4cbf388"
server_key_version: "encryptionkey@0"
I20260928 04:43:08.841082 2469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:08.852582 2469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:08.854802 2469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:08.854961 2469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:08.859252 2469 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:40145
I20260928 04:43:08.859371 2536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:40145 every 8 connection(s)
I20260928 04:43:08.859687 2469 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:43:08.860586 2537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:08.862876 2537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d033df4fa9f94e8f9a95c19fbb1795b0: Bootstrap starting.
I20260928 04:43:08.863420 2537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d033df4fa9f94e8f9a95c19fbb1795b0: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:08.863663 2537 log.cc:824] T 00000000000000000000000000000000 P d033df4fa9f94e8f9a95c19fbb1795b0: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:08.864375 2537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d033df4fa9f94e8f9a95c19fbb1795b0: No bootstrap required, opened a new log
I20260928 04:43:08.865962 2537 raft_consensus.cc:348] T 00000000000000000000000000000000 P d033df4fa9f94e8f9a95c19fbb1795b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d033df4fa9f94e8f9a95c19fbb1795b0" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 40145 } }
I20260928 04:43:08.866110 2537 raft_consensus.cc:374] T 00000000000000000000000000000000 P d033df4fa9f94e8f9a95c19fbb1795b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:08.866140 2537 raft_consensus.cc:729] T 00000000000000000000000000000000 P d033df4fa9f94e8f9a95c19fbb1795b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d033df4fa9f94e8f9a95c19fbb1795b0, State: Initialized, Role: FOLLOWER
I20260928 04:43:08.866292 2537 consensus_queue.cc:260] T 00000000000000000000000000000000 P d033df4fa9f94e8f9a95c19fbb1795b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d033df4fa9f94e8f9a95c19fbb1795b0" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 40145 } }
I20260928 04:43:08.866376 2537 raft_consensus.cc:388] T 00000000000000000000000000000000 P d033df4fa9f94e8f9a95c19fbb1795b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:43:08.866400 2537 raft_consensus.cc:482] T 00000000000000000000000000000000 P d033df4fa9f94e8f9a95c19fbb1795b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:43:08.866431 2537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d033df4fa9f94e8f9a95c19fbb1795b0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:08.866953 2537 raft_consensus.cc:504] T 00000000000000000000000000000000 P d033df4fa9f94e8f9a95c19fbb1795b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d033df4fa9f94e8f9a95c19fbb1795b0" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 40145 } }
I20260928 04:43:08.867041 2537 leader_election.cc:302] T 00000000000000000000000000000000 P d033df4fa9f94e8f9a95c19fbb1795b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d033df4fa9f94e8f9a95c19fbb1795b0; no voters:
I20260928 04:43:08.867182 2537 leader_election.cc:288] T 00000000000000000000000000000000 P d033df4fa9f94e8f9a95c19fbb1795b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:43:08.867255 2540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d033df4fa9f94e8f9a95c19fbb1795b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:08.867444 2540 raft_consensus.cc:686] T 00000000000000000000000000000000 P d033df4fa9f94e8f9a95c19fbb1795b0 [term 1 LEADER]: Becoming Leader. State: Replica: d033df4fa9f94e8f9a95c19fbb1795b0, State: Running, Role: LEADER
I20260928 04:43:08.867468 2537 sys_catalog.cc:564] T 00000000000000000000000000000000 P d033df4fa9f94e8f9a95c19fbb1795b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:43:08.867534 2540 consensus_queue.cc:237] T 00000000000000000000000000000000 P d033df4fa9f94e8f9a95c19fbb1795b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d033df4fa9f94e8f9a95c19fbb1795b0" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 40145 } }
I20260928 04:43:08.867799 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 2469
I20260928 04:43:08.867878 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:43:08.868062 1931 external_mini_cluster.cc:1428] Setting key d144de3bcdf5114eb21b18d6f59104fb
I20260928 04:43:08.868700 2541 sys_catalog.cc:455] T 00000000000000000000000000000000 P d033df4fa9f94e8f9a95c19fbb1795b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d033df4fa9f94e8f9a95c19fbb1795b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d033df4fa9f94e8f9a95c19fbb1795b0" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 40145 } } }
I20260928 04:43:08.868793 2541 sys_catalog.cc:458] T 00000000000000000000000000000000 P d033df4fa9f94e8f9a95c19fbb1795b0 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:08.869024 2542 sys_catalog.cc:455] T 00000000000000000000000000000000 P d033df4fa9f94e8f9a95c19fbb1795b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d033df4fa9f94e8f9a95c19fbb1795b0. Latest consensus state: current_term: 1 leader_uuid: "d033df4fa9f94e8f9a95c19fbb1795b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d033df4fa9f94e8f9a95c19fbb1795b0" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 40145 } } }
I20260928 04:43:08.869225 2558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:43:08.869233 2542 sys_catalog.cc:458] T 00000000000000000000000000000000 P d033df4fa9f94e8f9a95c19fbb1795b0 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:08.869794 2558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:43:08.871361 2558 catalog_manager.cc:1347] Generated new cluster ID: db14e39122a24f0ab1e34156e280bc74
I20260928 04:43:08.871412 2558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:43:08 dist-test-slave-j5c3 krb5kdc[2456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:08.880492 2556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:08.868768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58498 (local address 127.1.226.254:40145)
0928 04:43:08.868923 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:08.868934 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:08.868969 (+ 35us) server_negotiation.cc:407] Connection header received
0928 04:43:08.869096 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:08.869104 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:08.869177 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:08.869324 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:08.870236 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:08.871604 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:08.872710 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:08.873388 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:08.873512 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:08.876515 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:08.876521 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:08.876521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:08.876535 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:08.878581 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:08.879065 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:08.879068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:08.879069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:08.879208 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:08.879435 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:08.879435 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:08.879436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:08.879772 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:08.879929 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:08.880065 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 04:43:08.880239 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":62,"threads_started":1}
I20260928 04:43:08.889951 2558 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:43:08.890539 2558 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:43:08.897372 2558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d033df4fa9f94e8f9a95c19fbb1795b0: Generated new TSK 0
I20260928 04:43:08.897607 2558 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.226.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
I20260928 04:43:08.926750 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:0
--local_ip_for_outbound_sockets=127.1.226.193
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:40145
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:09.015516 2563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:09.015766 2563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:09.015793 2563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:09.017076 2563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:09.017122 2563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:09.017150 2563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:43:09.018601 2563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:40145
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:09.018813 2563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:09.018994 2563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:09.024363 2568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:09.024417 2569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:09.024363 2571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:09.024441 2563 server_base.cc:1034] running on GCE node
I20260928 04:43:09.024825 2563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:09.025059 2563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:09.026263 2563 hybrid_clock.cc:648] HybridClock initialized: now 1790570589026238 us; error 41 us; skew 500 ppm
Sep 28 04:43:09 dist-test-slave-j5c3 krb5kdc[2456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570589, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:09.029171 2563 init.cc:377] Logged in from keytab as kudu/127.1.226.193@KRBTEST.COM (short username kudu)
I20260928 04:43:09.029716 2563 webserver.cc:460] Webserver started at http://127.1.226.193:34533/ using document root <none> and password file <none>
I20260928 04:43:09.029970 2563 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:09.030035 2563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:09.030182 2563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:09.031270 2563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-0/data/instance:
uuid: "30c7e99e34e6400f913fc4fc4fbb716f"
format_stamp: "Formatted at 2026-09-28 04:43:09 on dist-test-slave-j5c3"
server_key: "8f94ddd4c8591a9c346ca63642b2f5df"
server_key_iv: "cf04ac99482993f02b32f8ac22ecbb90"
server_key_version: "encryptionkey@0"
I20260928 04:43:09.031648 2563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance:
uuid: "30c7e99e34e6400f913fc4fc4fbb716f"
format_stamp: "Formatted at 2026-09-28 04:43:09 on dist-test-slave-j5c3"
server_key: "8f94ddd4c8591a9c346ca63642b2f5df"
server_key_iv: "cf04ac99482993f02b32f8ac22ecbb90"
server_key_version: "encryptionkey@0"
I20260928 04:43:09.033077 2563 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:43:09.033854 2578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:09.034016 2563 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:43:09.034116 2563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "30c7e99e34e6400f913fc4fc4fbb716f"
format_stamp: "Formatted at 2026-09-28 04:43:09 on dist-test-slave-j5c3"
server_key: "8f94ddd4c8591a9c346ca63642b2f5df"
server_key_iv: "cf04ac99482993f02b32f8ac22ecbb90"
server_key_version: "encryptionkey@0"
I20260928 04:43:09.034217 2563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:09.069561 2563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:09.071852 2563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:09.072022 2563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:09.072306 2563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:09.072685 2563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:09.072738 2563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:09.072775 2563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:09.072817 2563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:09.078536 2563 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:33767
I20260928 04:43:09.078581 2691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:33767 every 8 connection(s)
I20260928 04:43:09.078900 2563 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
I20260928 04:43:09.082779 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 2563
I20260928 04:43:09.082885 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance
I20260928 04:43:09.083033 1931 external_mini_cluster.cc:1428] Setting key a5bef7fee27330b61e468c1c6898dff5
Sep 28 04:43:09 dist-test-slave-j5c3 krb5kdc[2456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570589, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:09.088940 2556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:09.079620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:54415 (local address 127.1.226.254:40145)
0928 04:43:09.079682 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:09.079684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:09.079904 (+ 220us) server_negotiation.cc:407] Connection header received
0928 04:43:09.081062 (+ 1158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:09.081063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:09.081074 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:09.081130 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:09.081715 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.082232 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.083348 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.083555 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.083637 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:09.085761 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:09.085763 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:09.085763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:09.085771 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:09.087455 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:09.087848 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:09.087849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:09.087850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:09.087920 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:09.088195 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:09.088196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:09.088196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:09.088361 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:09.088441 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:09.088694 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 04:43:09.088803 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 04:43:09.088961 2694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:09.079607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:40145 (local address 127.1.226.193:54415)
0928 04:43:09.079778 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:09.079820 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:09.080967 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:09.081188 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:09.081191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:09.081450 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:09.081644 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:09.081648 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.082316 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.082316 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:09.083256 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:09.083258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.083644 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.083645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:09.083765 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:09.084337 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:09.084345 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:09.085672 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:09.087534 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:09.087539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:09.087541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:09.087759 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:09.087986 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:09.087987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:09.087987 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:09.088093 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:09.088430 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:09.088435 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:09.088557 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 04:43:09.088645 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 04:43:09.088768 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
I20260928 04:43:09.089509 2692 heartbeater.cc:347] Connected to a master server at 127.1.226.254:40145
I20260928 04:43:09.089627 2692 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:09.089803 2692 heartbeater.cc:511] Master 127.1.226.254:40145 requested a full tablet report, sending...
I20260928 04:43:09.090337 2501 ts_manager.cc:194] Registered new tserver with Master: 30c7e99e34e6400f913fc4fc4fbb716f (127.1.226.193:33767)
I20260928 04:43:09.090958 2501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.193@KRBTEST.COM'} at 127.1.226.193:54415
WARNING: no policy specified for kudu/127.1.226.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
I20260928 04:43:09.110911 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:0
--local_ip_for_outbound_sockets=127.1.226.194
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:40145
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:09.192625 2697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:09.192847 2697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:09.192870 2697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:09.194092 2697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:09.194144 2697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:09.194175 2697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:43:09.195935 2697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:40145
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:09.196236 2697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:09.196484 2697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:09.201829 2703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:09.201843 2702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:09.201828 2705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:09.202183 2697 server_base.cc:1034] running on GCE node
I20260928 04:43:09.202430 2697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:09.202646 2697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:09.203830 2697 hybrid_clock.cc:648] HybridClock initialized: now 1790570589203807 us; error 32 us; skew 500 ppm
Sep 28 04:43:09 dist-test-slave-j5c3 krb5kdc[2456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570589, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:09.206765 2697 init.cc:377] Logged in from keytab as kudu/127.1.226.194@KRBTEST.COM (short username kudu)
I20260928 04:43:09.207309 2697 webserver.cc:460] Webserver started at http://127.1.226.194:44325/ using document root <none> and password file <none>
I20260928 04:43:09.207564 2697 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:09.207630 2697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:09.207775 2697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:09.208899 2697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-1/data/instance:
uuid: "6666d44d834a4e8dae9874c6b2305a69"
format_stamp: "Formatted at 2026-09-28 04:43:09 on dist-test-slave-j5c3"
server_key: "407c8dca3f46295bd56195157e8e9c71"
server_key_iv: "c0f390bc56b6c32ea09a229305b9cdc1"
server_key_version: "encryptionkey@0"
I20260928 04:43:09.209260 2697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance:
uuid: "6666d44d834a4e8dae9874c6b2305a69"
format_stamp: "Formatted at 2026-09-28 04:43:09 on dist-test-slave-j5c3"
server_key: "407c8dca3f46295bd56195157e8e9c71"
server_key_iv: "c0f390bc56b6c32ea09a229305b9cdc1"
server_key_version: "encryptionkey@0"
I20260928 04:43:09.210655 2697 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 04:43:09.211494 2712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:09.211688 2697 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:09.211796 2697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "6666d44d834a4e8dae9874c6b2305a69"
format_stamp: "Formatted at 2026-09-28 04:43:09 on dist-test-slave-j5c3"
server_key: "407c8dca3f46295bd56195157e8e9c71"
server_key_iv: "c0f390bc56b6c32ea09a229305b9cdc1"
server_key_version: "encryptionkey@0"
I20260928 04:43:09.211879 2697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:09.223168 2697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:09.226069 2697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:09.226233 2697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:09.226500 2697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:09.226864 2697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:09.226907 2697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:09.226938 2697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:09.226955 2697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:09.232580 2697 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:44929
I20260928 04:43:09.232734 2825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:44929 every 8 connection(s)
I20260928 04:43:09.232998 2697 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
I20260928 04:43:09.235666 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 2697
I20260928 04:43:09.235764 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance
I20260928 04:43:09.235924 1931 external_mini_cluster.cc:1428] Setting key 6a56a7e0156c0371ff4bbf3f54a4b65b
Sep 28 04:43:09 dist-test-slave-j5c3 krb5kdc[2456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570589, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:09.243898 2556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:09.233738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:55535 (local address 127.1.226.254:40145)
0928 04:43:09.233805 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:09.233807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:09.234028 (+ 221us) server_negotiation.cc:407] Connection header received
0928 04:43:09.234951 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:09.234952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:09.234963 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:09.235021 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:09.235708 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.236408 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.237564 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.237779 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.237870 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:09.240600 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:09.240602 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:09.240602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:09.240616 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:09.242130 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:09.242658 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:09.242659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:09.242659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:09.242757 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:09.243081 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:09.243082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:09.243083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:09.243228 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:09.243303 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:09.243733 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 04:43:09.243820 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 04:43:09.244192 2828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:09.233764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:40145 (local address 127.1.226.194:55535)
0928 04:43:09.233924 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:09.233960 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:09.234849 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:09.235075 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:09.235078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:09.235335 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:09.235571 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:09.235577 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.236499 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.236500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:09.237473 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:09.237476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.237879 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.237880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:09.238049 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:09.238627 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:09.238643 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:09.240453 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:09.242225 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:09.242233 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:09.242236 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:09.242575 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:09.242828 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:09.242829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:09.242830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:09.242985 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:09.243317 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:09.243324 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:09.243553 (+ 229us) client_negotiation.cc:769] Sending connection context
0928 04:43:09.243688 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 04:43:09.243888 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
WARNING: no policy specified for kudu/127.1.226.195@KRBTEST.COM; defaulting to no policy
I20260928 04:43:09.244906 2826 heartbeater.cc:347] Connected to a master server at 127.1.226.254:40145
I20260928 04:43:09.245018 2826 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:09.245178 2826 heartbeater.cc:511] Master 127.1.226.254:40145 requested a full tablet report, sending...
I20260928 04:43:09.245616 2501 ts_manager.cc:194] Registered new tserver with Master: 6666d44d834a4e8dae9874c6b2305a69 (127.1.226.194:44929)
I20260928 04:43:09.246059 2501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.194@KRBTEST.COM'} at 127.1.226.194:55535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
I20260928 04:43:09.262599 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:0
--local_ip_for_outbound_sockets=127.1.226.195
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:40145
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:09.342878 2831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:09.343086 2831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:09.343109 2831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:09.344355 2831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:09.344395 2831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:09.344424 2831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:43:09.345898 2831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:40145
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:09.346148 2831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:09.346387 2831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:09.351667 2836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:09.351660 2839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:09.352016 2831 server_base.cc:1034] running on GCE node
W20260928 04:43:09.352120 2837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:09.352411 2831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:09.352645 2831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:09.353811 2831 hybrid_clock.cc:648] HybridClock initialized: now 1790570589353800 us; error 29 us; skew 500 ppm
Sep 28 04:43:09 dist-test-slave-j5c3 krb5kdc[2456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570589, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:09.356428 2831 init.cc:377] Logged in from keytab as kudu/127.1.226.195@KRBTEST.COM (short username kudu)
I20260928 04:43:09.356911 2831 webserver.cc:460] Webserver started at http://127.1.226.195:42509/ using document root <none> and password file <none>
I20260928 04:43:09.357148 2831 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:09.357211 2831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:09.357363 2831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:09.358407 2831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-2/data/instance:
uuid: "27e4c768f3b947e9a9f93a093caa856c"
format_stamp: "Formatted at 2026-09-28 04:43:09 on dist-test-slave-j5c3"
server_key: "64b6dfaafd18635390ef5d80ed3e7028"
server_key_iv: "f1a597326b4973a706b4333cf9889940"
server_key_version: "encryptionkey@0"
I20260928 04:43:09.358769 2831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance:
uuid: "27e4c768f3b947e9a9f93a093caa856c"
format_stamp: "Formatted at 2026-09-28 04:43:09 on dist-test-slave-j5c3"
server_key: "64b6dfaafd18635390ef5d80ed3e7028"
server_key_iv: "f1a597326b4973a706b4333cf9889940"
server_key_version: "encryptionkey@0"
I20260928 04:43:09.360087 2831 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:43:09.360869 2846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:09.361063 2831 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:43:09.361155 2831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "27e4c768f3b947e9a9f93a093caa856c"
format_stamp: "Formatted at 2026-09-28 04:43:09 on dist-test-slave-j5c3"
server_key: "64b6dfaafd18635390ef5d80ed3e7028"
server_key_iv: "f1a597326b4973a706b4333cf9889940"
server_key_version: "encryptionkey@0"
I20260928 04:43:09.361234 2831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:09.373948 2831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:09.376299 2831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:09.376451 2831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:09.376668 2831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:09.376965 2831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:09.377014 2831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:09.377049 2831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:09.377076 2831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:09.382746 2831 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:39807
I20260928 04:43:09.382790 2959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:39807 every 8 connection(s)
I20260928 04:43:09.383438 2831 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
I20260928 04:43:09.387327 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 2831
I20260928 04:43:09.387430 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance
I20260928 04:43:09.387584 1931 external_mini_cluster.cc:1428] Setting key 4e9cf580d7324979bac577aac7145a02
Sep 28 04:43:09 dist-test-slave-j5c3 krb5kdc[2456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570589, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:09.393519 2556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:09.384264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:32961 (local address 127.1.226.254:40145)
0928 04:43:09.384322 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:09.384324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:09.384550 (+ 226us) server_negotiation.cc:407] Connection header received
0928 04:43:09.385697 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:09.385700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:09.385717 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:09.385801 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:09.386555 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.387111 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.388597 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.388783 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.388845 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:09.390970 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:09.390971 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:09.390972 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:09.390978 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:09.392362 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:09.392754 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:09.392755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:09.392756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:09.392825 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:09.393026 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:09.393026 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:09.393027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:09.393125 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:09.393173 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:09.393395 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 04:43:09.393453 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 04:43:09.393774 2962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:09.384300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:40145 (local address 127.1.226.195:32961)
0928 04:43:09.384434 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:09.384469 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:09.385521 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:09.385901 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:09.385904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:09.386259 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:09.386486 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:09.386490 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.387197 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.387198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:09.388533 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:09.388535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.388855 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.388856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:09.388959 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:09.389422 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:09.389430 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:09.390881 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:09.392443 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:09.392449 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:09.392451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:09.392657 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:09.392876 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:09.392877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:09.392877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:09.392977 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:09.393173 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:09.393177 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:09.393293 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 04:43:09.393373 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 04:43:09.393487 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
I20260928 04:43:09.394250 2960 heartbeater.cc:347] Connected to a master server at 127.1.226.254:40145
I20260928 04:43:09.394342 2960 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:09.394521 2960 heartbeater.cc:511] Master 127.1.226.254:40145 requested a full tablet report, sending...
I20260928 04:43:09.394874 2501 ts_manager.cc:194] Registered new tserver with Master: 27e4c768f3b947e9a9f93a093caa856c (127.1.226.195:39807)
I20260928 04:43:09.395206 2501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.195@KRBTEST.COM'} at 127.1.226.195:32961
I20260928 04:43:09.399165 1931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:43:09.405962 2556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:09.400684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58514 (local address 127.1.226.254:40145)
0928 04:43:09.400724 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:09.400726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:09.400819 (+ 93us) server_negotiation.cc:407] Connection header received
0928 04:43:09.400871 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:09.400871 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:09.400879 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:09.400923 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:09.401411 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.401859 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.402525 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.402694 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.402739 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:09.403802 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:09.403803 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:09.403804 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:09.403809 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:09.405091 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:09.405378 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:09.405379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:09.405379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:09.405424 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:09.405565 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:09.405566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:09.405566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:09.405679 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:09.405745 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:09.405824 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 04:43:09.405877 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:43:09.408303 2501 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:43:09.409875 2501 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:43:09.416450 2972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:09.412223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:33767 (local address 127.0.0.1:56234)
0928 04:43:09.412400 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:09.412442 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:09.412528 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:09.412843 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:09.412846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:09.412848 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:09.412919 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:09.412921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.413887 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.413888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:09.415127 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:09.415130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.416092 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.416093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:09.416209 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:09.416226 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 04:43:09.416307 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 04:43:09.416367 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":56,"threads_started":1}
I20260928 04:43:09.416458 2973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:09.412306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56234 (local address 127.1.226.193:33767)
0928 04:43:09.412662 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:09.412667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:09.412677 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:43:09.412719 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:09.412721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:09.412739 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:09.412997 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:09.413076 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.413780 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.415223 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.416018 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.416091 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:09.416173 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:09.416332 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 04:43:09.416378 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":73,"threads_started":1}
I20260928 04:43:09.416841 2971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:09.412331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:39807 (local address 127.0.0.1:39382)
0928 04:43:09.412704 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:09.412715 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:09.412773 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:09.413020 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:09.413020 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:09.413022 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:09.413057 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:09.413059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.414314 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.414315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:09.415329 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:09.415330 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.416421 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.416422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:09.416488 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:09.416493 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:43:09.416728 (+ 235us) client_negotiation.cc:240] Negotiation successful
0928 04:43:09.416771 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":62,"threads_started":1}
I20260928 04:43:09.416862 2975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:09.412470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39382 (local address 127.1.226.195:39807)
0928 04:43:09.412662 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:09.412667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:09.412787 (+ 120us) server_negotiation.cc:407] Connection header received
0928 04:43:09.412852 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:09.412855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:09.412869 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:09.412939 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:09.413132 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.414198 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.415400 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.416252 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.416336 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:09.416388 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:09.416748 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 04:43:09.416790 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":86,"threads_started":1}
I20260928 04:43:09.417394 2974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:09.412240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:44929 (local address 127.0.0.1:54666)
0928 04:43:09.412441 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:09.412452 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:09.412517 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:09.412955 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:09.412956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:09.412958 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:09.413011 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:09.413013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.415395 (+ 2382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.415396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:09.416262 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:09.416264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.417143 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.417144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:09.417211 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:09.417216 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:43:09.417278 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 04:43:09.417318 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":73,"threads_started":1}
I20260928 04:43:09.417408 2976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:09.412489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54666 (local address 127.1.226.194:44929)
0928 04:43:09.412673 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:09.412677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:09.412723 (+ 46us) server_negotiation.cc:407] Connection header received
0928 04:43:09.412770 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:09.412773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:09.412788 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:09.412867 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:09.413099 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.414207 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.416358 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.417059 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.417139 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:09.417190 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:09.417291 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 04:43:09.417336 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":78,"threads_started":1}
I20260928 04:43:09.417522 2626 tablet_service.cc:1467] Processing CreateTablet for tablet 23c60143ad864e6eb018afa8c4faae38 (DEFAULT_TABLE table=test-table [id=27ae360442b34d36bc6d369de7b95c4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:09.417886 2626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23c60143ad864e6eb018afa8c4faae38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:09.418036 2760 tablet_service.cc:1467] Processing CreateTablet for tablet 23c60143ad864e6eb018afa8c4faae38 (DEFAULT_TABLE table=test-table [id=27ae360442b34d36bc6d369de7b95c4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:09.418358 2760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23c60143ad864e6eb018afa8c4faae38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:09.420675 2977 tablet_bootstrap.cc:492] T 23c60143ad864e6eb018afa8c4faae38 P 30c7e99e34e6400f913fc4fc4fbb716f: Bootstrap starting.
I20260928 04:43:09.421293 2978 tablet_bootstrap.cc:492] T 23c60143ad864e6eb018afa8c4faae38 P 6666d44d834a4e8dae9874c6b2305a69: Bootstrap starting.
I20260928 04:43:09.421418 2894 tablet_service.cc:1467] Processing CreateTablet for tablet 23c60143ad864e6eb018afa8c4faae38 (DEFAULT_TABLE table=test-table [id=27ae360442b34d36bc6d369de7b95c4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:09.421541 2977 tablet_bootstrap.cc:654] T 23c60143ad864e6eb018afa8c4faae38 P 30c7e99e34e6400f913fc4fc4fbb716f: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:09.421733 2894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23c60143ad864e6eb018afa8c4faae38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:09.422309 2978 tablet_bootstrap.cc:654] T 23c60143ad864e6eb018afa8c4faae38 P 6666d44d834a4e8dae9874c6b2305a69: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:09.422467 2977 log.cc:824] T 23c60143ad864e6eb018afa8c4faae38 P 30c7e99e34e6400f913fc4fc4fbb716f: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:09.422883 2978 log.cc:824] T 23c60143ad864e6eb018afa8c4faae38 P 6666d44d834a4e8dae9874c6b2305a69: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:09.423282 2977 tablet_bootstrap.cc:492] T 23c60143ad864e6eb018afa8c4faae38 P 30c7e99e34e6400f913fc4fc4fbb716f: No bootstrap required, opened a new log
I20260928 04:43:09.423372 2977 ts_tablet_manager.cc:1397] T 23c60143ad864e6eb018afa8c4faae38 P 30c7e99e34e6400f913fc4fc4fbb716f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 04:43:09.423586 2978 tablet_bootstrap.cc:492] T 23c60143ad864e6eb018afa8c4faae38 P 6666d44d834a4e8dae9874c6b2305a69: No bootstrap required, opened a new log
I20260928 04:43:09.423647 2978 ts_tablet_manager.cc:1397] T 23c60143ad864e6eb018afa8c4faae38 P 6666d44d834a4e8dae9874c6b2305a69: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:43:09.424978 2981 tablet_bootstrap.cc:492] T 23c60143ad864e6eb018afa8c4faae38 P 27e4c768f3b947e9a9f93a093caa856c: Bootstrap starting.
I20260928 04:43:09.425484 2978 raft_consensus.cc:348] T 23c60143ad864e6eb018afa8c4faae38 P 6666d44d834a4e8dae9874c6b2305a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c7e99e34e6400f913fc4fc4fbb716f" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 33767 } } peers { permanent_uuid: "27e4c768f3b947e9a9f93a093caa856c" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39807 } } peers { permanent_uuid: "6666d44d834a4e8dae9874c6b2305a69" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44929 } }
I20260928 04:43:09.425685 2978 raft_consensus.cc:374] T 23c60143ad864e6eb018afa8c4faae38 P 6666d44d834a4e8dae9874c6b2305a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:09.425730 2978 raft_consensus.cc:729] T 23c60143ad864e6eb018afa8c4faae38 P 6666d44d834a4e8dae9874c6b2305a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6666d44d834a4e8dae9874c6b2305a69, State: Initialized, Role: FOLLOWER
I20260928 04:43:09.425740 2981 tablet_bootstrap.cc:654] T 23c60143ad864e6eb018afa8c4faae38 P 27e4c768f3b947e9a9f93a093caa856c: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:09.425618 2977 raft_consensus.cc:348] T 23c60143ad864e6eb018afa8c4faae38 P 30c7e99e34e6400f913fc4fc4fbb716f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c7e99e34e6400f913fc4fc4fbb716f" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 33767 } } peers { permanent_uuid: "27e4c768f3b947e9a9f93a093caa856c" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39807 } } peers { permanent_uuid: "6666d44d834a4e8dae9874c6b2305a69" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44929 } }
I20260928 04:43:09.425789 2977 raft_consensus.cc:374] T 23c60143ad864e6eb018afa8c4faae38 P 30c7e99e34e6400f913fc4fc4fbb716f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:09.425822 2977 raft_consensus.cc:729] T 23c60143ad864e6eb018afa8c4faae38 P 30c7e99e34e6400f913fc4fc4fbb716f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c7e99e34e6400f913fc4fc4fbb716f, State: Initialized, Role: FOLLOWER
I20260928 04:43:09.425896 2978 consensus_queue.cc:260] T 23c60143ad864e6eb018afa8c4faae38 P 6666d44d834a4e8dae9874c6b2305a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c7e99e34e6400f913fc4fc4fbb716f" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 33767 } } peers { permanent_uuid: "27e4c768f3b947e9a9f93a093caa856c" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39807 } } peers { permanent_uuid: "6666d44d834a4e8dae9874c6b2305a69" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44929 } }
I20260928 04:43:09.425959 2977 consensus_queue.cc:260] T 23c60143ad864e6eb018afa8c4faae38 P 30c7e99e34e6400f913fc4fc4fbb716f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c7e99e34e6400f913fc4fc4fbb716f" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 33767 } } peers { permanent_uuid: "27e4c768f3b947e9a9f93a093caa856c" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39807 } } peers { permanent_uuid: "6666d44d834a4e8dae9874c6b2305a69" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44929 } }
I20260928 04:43:09.426254 2692 heartbeater.cc:503] Master 127.1.226.254:40145 was elected leader, sending a full tablet report...
I20260928 04:43:09.426252 2978 ts_tablet_manager.cc:1428] T 23c60143ad864e6eb018afa8c4faae38 P 6666d44d834a4e8dae9874c6b2305a69: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:43:09.426316 2981 log.cc:824] T 23c60143ad864e6eb018afa8c4faae38 P 27e4c768f3b947e9a9f93a093caa856c: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:09.426367 2826 heartbeater.cc:503] Master 127.1.226.254:40145 was elected leader, sending a full tablet report...
I20260928 04:43:09.426522 2977 ts_tablet_manager.cc:1428] T 23c60143ad864e6eb018afa8c4faae38 P 30c7e99e34e6400f913fc4fc4fbb716f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:43:09.427611 2981 tablet_bootstrap.cc:492] T 23c60143ad864e6eb018afa8c4faae38 P 27e4c768f3b947e9a9f93a093caa856c: No bootstrap required, opened a new log
I20260928 04:43:09.427703 2981 ts_tablet_manager.cc:1397] T 23c60143ad864e6eb018afa8c4faae38 P 27e4c768f3b947e9a9f93a093caa856c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 04:43:09.429209 2981 raft_consensus.cc:348] T 23c60143ad864e6eb018afa8c4faae38 P 27e4c768f3b947e9a9f93a093caa856c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c7e99e34e6400f913fc4fc4fbb716f" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 33767 } } peers { permanent_uuid: "27e4c768f3b947e9a9f93a093caa856c" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39807 } } peers { permanent_uuid: "6666d44d834a4e8dae9874c6b2305a69" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44929 } }
I20260928 04:43:09.429386 2981 raft_consensus.cc:374] T 23c60143ad864e6eb018afa8c4faae38 P 27e4c768f3b947e9a9f93a093caa856c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:09.429433 2981 raft_consensus.cc:729] T 23c60143ad864e6eb018afa8c4faae38 P 27e4c768f3b947e9a9f93a093caa856c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27e4c768f3b947e9a9f93a093caa856c, State: Initialized, Role: FOLLOWER
I20260928 04:43:09.429558 2981 consensus_queue.cc:260] T 23c60143ad864e6eb018afa8c4faae38 P 27e4c768f3b947e9a9f93a093caa856c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c7e99e34e6400f913fc4fc4fbb716f" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 33767 } } peers { permanent_uuid: "27e4c768f3b947e9a9f93a093caa856c" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39807 } } peers { permanent_uuid: "6666d44d834a4e8dae9874c6b2305a69" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44929 } }
I20260928 04:43:09.429819 2981 ts_tablet_manager.cc:1428] T 23c60143ad864e6eb018afa8c4faae38 P 27e4c768f3b947e9a9f93a093caa856c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:43:09.429864 2960 heartbeater.cc:503] Master 127.1.226.254:40145 was elected leader, sending a full tablet report...
W20260928 04:43:09.483606 2827 tablet.cc:2369] T 23c60143ad864e6eb018afa8c4faae38 P 6666d44d834a4e8dae9874c6b2305a69: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:43:09.552277 2985 raft_consensus.cc:482] T 23c60143ad864e6eb018afa8c4faae38 P 27e4c768f3b947e9a9f93a093caa856c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:43:09.552444 2985 raft_consensus.cc:504] T 23c60143ad864e6eb018afa8c4faae38 P 27e4c768f3b947e9a9f93a093caa856c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c7e99e34e6400f913fc4fc4fbb716f" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 33767 } } peers { permanent_uuid: "27e4c768f3b947e9a9f93a093caa856c" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39807 } } peers { permanent_uuid: "6666d44d834a4e8dae9874c6b2305a69" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44929 } }
I20260928 04:43:09.552945 2985 leader_election.cc:288] T 23c60143ad864e6eb018afa8c4faae38 P 27e4c768f3b947e9a9f93a093caa856c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30c7e99e34e6400f913fc4fc4fbb716f (127.1.226.193:33767), 6666d44d834a4e8dae9874c6b2305a69 (127.1.226.194:44929)
I20260928 04:43:09.557054 2976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:09.553268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:59399 (local address 127.1.226.194:44929)
0928 04:43:09.553319 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:09.553321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:09.553517 (+ 196us) server_negotiation.cc:407] Connection header received
0928 04:43:09.553555 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:09.553557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:09.553567 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:09.553637 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:09.553760 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.554509 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.556084 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.556696 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.556781 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:09.556791 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:09.556941 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 04:43:09.556977 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 04:43:09.557099 2986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:09.553173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:44929 (local address 127.1.226.195:59399)
0928 04:43:09.553430 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:09.553453 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:09.553502 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:09.553639 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:09.553640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:09.553642 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:09.553697 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:09.553698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.554591 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.554592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:09.555933 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:09.555936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.556779 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.556780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:09.556859 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:09.556862 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:43:09.556932 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 04:43:09.556976 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":145,"thread_start_us":125,"threads_started":1}
I20260928 04:43:09.557226 2973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:09.553229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:35069 (local address 127.1.226.193:33767)
0928 04:43:09.553282 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:09.553284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:09.553298 (+ 14us) server_negotiation.cc:407] Connection header received
0928 04:43:09.553388 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:09.553389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:09.553402 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:09.553463 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:09.553639 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.554495 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.556046 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.556693 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.556799 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:09.556809 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:09.556988 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 04:43:09.557027 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 04:43:09.557366 2962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:09.553164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:33767 (local address 127.1.226.195:35069)
0928 04:43:09.553240 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:09.553254 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:09.553302 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:09.553482 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:09.553483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:09.553485 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:09.553558 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:09.553560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.554594 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.554595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:09.555933 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:09.555936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.556805 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.556806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:09.556898 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:09.556901 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:43:09.556981 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 04:43:09.557027 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":74}
I20260928 04:43:09.557485 2780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c60143ad864e6eb018afa8c4faae38" candidate_uuid: "27e4c768f3b947e9a9f93a093caa856c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6666d44d834a4e8dae9874c6b2305a69" is_pre_election: true
I20260928 04:43:09.557682 2780 raft_consensus.cc:2454] T 23c60143ad864e6eb018afa8c4faae38 P 6666d44d834a4e8dae9874c6b2305a69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27e4c768f3b947e9a9f93a093caa856c in term 0.
I20260928 04:43:09.557925 2848 leader_election.cc:302] T 23c60143ad864e6eb018afa8c4faae38 P 27e4c768f3b947e9a9f93a093caa856c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27e4c768f3b947e9a9f93a093caa856c, 6666d44d834a4e8dae9874c6b2305a69; no voters:
I20260928 04:43:09.557991 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c60143ad864e6eb018afa8c4faae38" candidate_uuid: "27e4c768f3b947e9a9f93a093caa856c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c7e99e34e6400f913fc4fc4fbb716f" is_pre_election: true
I20260928 04:43:09.558068 2985 raft_consensus.cc:2781] T 23c60143ad864e6eb018afa8c4faae38 P 27e4c768f3b947e9a9f93a093caa856c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:43:09.558117 2985 raft_consensus.cc:482] T 23c60143ad864e6eb018afa8c4faae38 P 27e4c768f3b947e9a9f93a093caa856c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:43:09.558171 2985 raft_consensus.cc:3037] T 23c60143ad864e6eb018afa8c4faae38 P 27e4c768f3b947e9a9f93a093caa856c [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:09.558168 2646 raft_consensus.cc:2454] T 23c60143ad864e6eb018afa8c4faae38 P 30c7e99e34e6400f913fc4fc4fbb716f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27e4c768f3b947e9a9f93a093caa856c in term 0.
I20260928 04:43:09.558996 2985 raft_consensus.cc:504] T 23c60143ad864e6eb018afa8c4faae38 P 27e4c768f3b947e9a9f93a093caa856c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c7e99e34e6400f913fc4fc4fbb716f" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 33767 } } peers { permanent_uuid: "27e4c768f3b947e9a9f93a093caa856c" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39807 } } peers { permanent_uuid: "6666d44d834a4e8dae9874c6b2305a69" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44929 } }
I20260928 04:43:09.559156 2985 leader_election.cc:288] T 23c60143ad864e6eb018afa8c4faae38 P 27e4c768f3b947e9a9f93a093caa856c [CANDIDATE]: Term 1 election: Requested vote from peers 30c7e99e34e6400f913fc4fc4fbb716f (127.1.226.193:33767), 6666d44d834a4e8dae9874c6b2305a69 (127.1.226.194:44929)
I20260928 04:43:09.559320 2780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c60143ad864e6eb018afa8c4faae38" candidate_uuid: "27e4c768f3b947e9a9f93a093caa856c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6666d44d834a4e8dae9874c6b2305a69"
I20260928 04:43:09.559324 2646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c60143ad864e6eb018afa8c4faae38" candidate_uuid: "27e4c768f3b947e9a9f93a093caa856c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c7e99e34e6400f913fc4fc4fbb716f"
I20260928 04:43:09.559405 2646 raft_consensus.cc:3037] T 23c60143ad864e6eb018afa8c4faae38 P 30c7e99e34e6400f913fc4fc4fbb716f [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:09.559412 2780 raft_consensus.cc:3037] T 23c60143ad864e6eb018afa8c4faae38 P 6666d44d834a4e8dae9874c6b2305a69 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:09.560093 2646 raft_consensus.cc:2454] T 23c60143ad864e6eb018afa8c4faae38 P 30c7e99e34e6400f913fc4fc4fbb716f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27e4c768f3b947e9a9f93a093caa856c in term 1.
I20260928 04:43:09.560199 2780 raft_consensus.cc:2454] T 23c60143ad864e6eb018afa8c4faae38 P 6666d44d834a4e8dae9874c6b2305a69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27e4c768f3b947e9a9f93a093caa856c in term 1.
I20260928 04:43:09.560254 2847 leader_election.cc:302] T 23c60143ad864e6eb018afa8c4faae38 P 27e4c768f3b947e9a9f93a093caa856c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27e4c768f3b947e9a9f93a093caa856c, 30c7e99e34e6400f913fc4fc4fbb716f; no voters:
I20260928 04:43:09.560348 2985 raft_consensus.cc:2781] T 23c60143ad864e6eb018afa8c4faae38 P 27e4c768f3b947e9a9f93a093caa856c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:09.560487 2985 raft_consensus.cc:686] T 23c60143ad864e6eb018afa8c4faae38 P 27e4c768f3b947e9a9f93a093caa856c [term 1 LEADER]: Becoming Leader. State: Replica: 27e4c768f3b947e9a9f93a093caa856c, State: Running, Role: LEADER
I20260928 04:43:09.560590 2985 consensus_queue.cc:237] T 23c60143ad864e6eb018afa8c4faae38 P 27e4c768f3b947e9a9f93a093caa856c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c7e99e34e6400f913fc4fc4fbb716f" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 33767 } } peers { permanent_uuid: "27e4c768f3b947e9a9f93a093caa856c" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39807 } } peers { permanent_uuid: "6666d44d834a4e8dae9874c6b2305a69" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44929 } }
I20260928 04:43:09.561450 2501 catalog_manager.cc:5495] T 23c60143ad864e6eb018afa8c4faae38 P 27e4c768f3b947e9a9f93a093caa856c reported cstate change: term changed from 0 to 1, leader changed from <none> to 27e4c768f3b947e9a9f93a093caa856c (127.1.226.195). New cstate: current_term: 1 leader_uuid: "27e4c768f3b947e9a9f93a093caa856c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c7e99e34e6400f913fc4fc4fbb716f" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 33767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27e4c768f3b947e9a9f93a093caa856c" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6666d44d834a4e8dae9874c6b2305a69" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44929 } health_report { overall_health: UNKNOWN } } }
W20260928 04:43:09.579650 2693 tablet.cc:2369] T 23c60143ad864e6eb018afa8c4faae38 P 30c7e99e34e6400f913fc4fc4fbb716f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 04:43:09 dist-test-slave-j5c3 krb5kdc[2456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.193@KRBTEST.COM
I20260928 04:43:09.601105 2973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:09.593535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56240 (local address 127.1.226.193:33767)
0928 04:43:09.593585 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:09.593587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:09.593671 (+ 84us) server_negotiation.cc:407] Connection header received
0928 04:43:09.593725 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:09.593726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:09.593735 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:09.593786 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:09.594480 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.594973 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.595686 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.595882 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.595931 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:09.598230 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:09.598239 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:09.598240 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:09.598246 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:09.599943 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:09.600275 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:09.600279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:09.600279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:09.600339 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:09.600593 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:09.600594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:09.600594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:09.600773 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:09.600881 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:09.600980 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 04:43:09.601039 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 04:43:09 dist-test-slave-j5c3 krb5kdc[2456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570589, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 04:43:09 dist-test-slave-j5c3 krb5kdc[2456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570589, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.226.193@KRBTEST.COM
I20260928 04:43:09.624464 2973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:09.616836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56256 (local address 127.1.226.193:33767)
0928 04:43:09.616902 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:09.616905 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:09.617399 (+ 494us) server_negotiation.cc:407] Connection header received
0928 04:43:09.617503 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:09.617504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:09.617515 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:09.617579 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:09.618219 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.618709 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.619434 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.619633 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.619722 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:09.622157 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:09.622159 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:09.622159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:09.622166 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:09.623388 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:09.623702 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:09.623703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:09.623703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:09.623758 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:09.623926 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:09.623926 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:09.623926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:09.624022 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:09.624101 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:09.624326 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 04:43:09.624394 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 04:43:09.634039 2961 tablet.cc:2369] T 23c60143ad864e6eb018afa8c4faae38 P 27e4c768f3b947e9a9f93a093caa856c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 04:43:09 dist-test-slave-j5c3 krb5kdc[2456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 04:43:09 dist-test-slave-j5c3 krb5kdc[2456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.193@KRBTEST.COM
I20260928 04:43:09.646595 2973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:09.639526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56262 (local address 127.1.226.193:33767)
0928 04:43:09.639588 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:09.639590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:09.639693 (+ 103us) server_negotiation.cc:407] Connection header received
0928 04:43:09.639764 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:09.639765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:09.639774 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:09.639823 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:09.640533 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.641001 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.641708 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:09.641887 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:09.641937 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:09.644239 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:09.644241 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:09.644241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:09.644249 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:09.645638 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:09.645980 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:09.645980 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:09.645981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:09.646043 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:09.646203 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:09.646204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:09.646204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:09.646301 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:09.646377 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:09.646469 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 04:43:09.646529 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 04:43:09.647058 2606 tablet_service.cc:2908] Rejecting scan request for tablet 23c60143ad864e6eb018afa8c4faae38: Uninitialized: clean time has not yet been initialized
I20260928 04:43:09.647861 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 2563
I20260928 04:43:09.652498 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 2697
I20260928 04:43:09.657156 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 2831
I20260928 04:43:09.661352 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 2469
2026-09-28T04:43:09Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2087 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:43:09 dist-test-slave-j5c3 krb5kdc[3001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:43:09 dist-test-slave-j5c3 krb5kdc[3001](info): set up 2 sockets
Sep 28 04:43:09 dist-test-slave-j5c3 krb5kdc[3001](info): commencing operation
krb5kdc: starting...
W20260928 04:43:10.748293 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:43:10 dist-test-slave-j5c3 krb5kdc[3001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:43:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:43:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:43:10.871269 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:45591
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:44349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:45591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:10.960073 3014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:10.960356 3014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:10.960399 3014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:10.961632 3014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:43:10.961707 3014 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 04:43:10.961732 3014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:10.961760 3014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:10.961789 3014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:43:10.961812 3014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:43:10.963399 3014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:44349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:45591
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:45591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:10.963693 3014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:10.963925 3014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:10.969058 3019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:10.969069 3022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:10.969138 3014 server_base.cc:1034] running on GCE node
W20260928 04:43:10.969076 3020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:10.969583 3014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:10.969831 3014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:10.971002 3014 hybrid_clock.cc:648] HybridClock initialized: now 1790570590970981 us; error 34 us; skew 500 ppm
Sep 28 04:43:10 dist-test-slave-j5c3 krb5kdc[3001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570590, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:10.979908 3014 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:43:10.980481 3014 webserver.cc:460] Webserver started at http://127.1.226.254:35301/ using document root <none> and password file <none>
I20260928 04:43:10.980719 3014 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:10.980786 3014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:10.980934 3014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:10.982059 3014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "2bec407195a84052afacd9c2624e3bc0"
format_stamp: "Formatted at 2026-09-28 04:43:10 on dist-test-slave-j5c3"
server_key: "d64024910f14fb6f041861291dc6fbcc"
server_key_iv: "2c0b18e361eba07ce082fe1d09d5e516"
server_key_version: "encryptionkey@0"
I20260928 04:43:10.982420 3014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "2bec407195a84052afacd9c2624e3bc0"
format_stamp: "Formatted at 2026-09-28 04:43:10 on dist-test-slave-j5c3"
server_key: "d64024910f14fb6f041861291dc6fbcc"
server_key_iv: "2c0b18e361eba07ce082fe1d09d5e516"
server_key_version: "encryptionkey@0"
I20260928 04:43:10.983845 3014 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:43:10.984779 3029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:10.984988 3014 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:43:10.985092 3014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "2bec407195a84052afacd9c2624e3bc0"
format_stamp: "Formatted at 2026-09-28 04:43:10 on dist-test-slave-j5c3"
server_key: "d64024910f14fb6f041861291dc6fbcc"
server_key_iv: "2c0b18e361eba07ce082fe1d09d5e516"
server_key_version: "encryptionkey@0"
I20260928 04:43:10.985163 3014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:10.998210 3014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:11.000823 3014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:11.001003 3014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:11.006465 3014 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:45591
I20260928 04:43:11.006515 3091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:45591 every 8 connection(s)
I20260928 04:43:11.006958 3014 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:43:11.007674 3092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:11.010366 3092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bec407195a84052afacd9c2624e3bc0: Bootstrap starting.
I20260928 04:43:11.010953 3092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bec407195a84052afacd9c2624e3bc0: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:11.011228 3092 log.cc:824] T 00000000000000000000000000000000 P 2bec407195a84052afacd9c2624e3bc0: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:11.011955 3092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bec407195a84052afacd9c2624e3bc0: No bootstrap required, opened a new log
I20260928 04:43:11.014353 3092 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bec407195a84052afacd9c2624e3bc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bec407195a84052afacd9c2624e3bc0" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45591 } }
I20260928 04:43:11.014513 3092 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bec407195a84052afacd9c2624e3bc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:11.014539 3092 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bec407195a84052afacd9c2624e3bc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bec407195a84052afacd9c2624e3bc0, State: Initialized, Role: FOLLOWER
I20260928 04:43:11.014654 3092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bec407195a84052afacd9c2624e3bc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bec407195a84052afacd9c2624e3bc0" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45591 } }
I20260928 04:43:11.014744 3092 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bec407195a84052afacd9c2624e3bc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:43:11.014787 3092 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bec407195a84052afacd9c2624e3bc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:43:11.014861 3092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bec407195a84052afacd9c2624e3bc0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:11.015470 3092 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bec407195a84052afacd9c2624e3bc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bec407195a84052afacd9c2624e3bc0" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45591 } }
I20260928 04:43:11.015594 3092 leader_election.cc:302] T 00000000000000000000000000000000 P 2bec407195a84052afacd9c2624e3bc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bec407195a84052afacd9c2624e3bc0; no voters:
I20260928 04:43:11.015789 3092 leader_election.cc:288] T 00000000000000000000000000000000 P 2bec407195a84052afacd9c2624e3bc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:43:11.015900 3095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bec407195a84052afacd9c2624e3bc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:11.016074 3092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bec407195a84052afacd9c2624e3bc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:43:11.016090 3095 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bec407195a84052afacd9c2624e3bc0 [term 1 LEADER]: Becoming Leader. State: Replica: 2bec407195a84052afacd9c2624e3bc0, State: Running, Role: LEADER
I20260928 04:43:11.016242 3095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bec407195a84052afacd9c2624e3bc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bec407195a84052afacd9c2624e3bc0" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45591 } }
I20260928 04:43:11.016685 3097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bec407195a84052afacd9c2624e3bc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bec407195a84052afacd9c2624e3bc0. Latest consensus state: current_term: 1 leader_uuid: "2bec407195a84052afacd9c2624e3bc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bec407195a84052afacd9c2624e3bc0" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45591 } } }
I20260928 04:43:11.016796 3097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bec407195a84052afacd9c2624e3bc0 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:11.016963 3096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bec407195a84052afacd9c2624e3bc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bec407195a84052afacd9c2624e3bc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bec407195a84052afacd9c2624e3bc0" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45591 } } }
I20260928 04:43:11.017033 3096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bec407195a84052afacd9c2624e3bc0 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:11.017051 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 3014
I20260928 04:43:11.017151 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:43:11.017236 3104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:43:11.017325 1931 external_mini_cluster.cc:1428] Setting key fc6a0ebb253ed1452e324b0337ecd1e6
I20260928 04:43:11.017810 3104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:43:11.019462 3104 catalog_manager.cc:1347] Generated new cluster ID: 5ded6de65dab4195a34ce535864d917a
I20260928 04:43:11.019523 3104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:43:11 dist-test-slave-j5c3 krb5kdc[3001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:11.030457 3112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.018034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51370 (local address 127.1.226.254:45591)
0928 04:43:11.018467 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:11.018478 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:11.018699 (+ 221us) server_negotiation.cc:407] Connection header received
0928 04:43:11.018875 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:11.018883 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:11.018966 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:11.019977 (+ 1011us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:11.020069 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.021524 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.022615 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.023235 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.023661 (+ 426us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.026383 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:11.026388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:11.026389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:11.026403 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:11.028648 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:11.029096 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:11.029099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:11.029099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:11.029220 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:11.029446 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:11.029447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:11.029447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:11.029779 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:11.029920 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:11.030050 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 04:43:11.030195 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":74,"threads_started":1}
I20260928 04:43:11.033179 3104 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:43:11.033910 3104 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:43:11.038223 3104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bec407195a84052afacd9c2624e3bc0: Generated new TSK 0
I20260928 04:43:11.038450 3104 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.226.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
I20260928 04:43:11.067483 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:0
--local_ip_for_outbound_sockets=127.1.226.193
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:45591
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:44349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:11.149243 3118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:11.149502 3118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:11.149541 3118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:11.150728 3118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:11.150794 3118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:11.150826 3118 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 04:43:11.150866 3118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:43:11.152379 3118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:44349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.226.254:45591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:11.152642 3118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:11.152865 3118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:11.157832 3124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:11.157832 3123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:11.158070 3118 server_base.cc:1034] running on GCE node
W20260928 04:43:11.157876 3126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:11.158384 3118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:11.158643 3118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:11.159812 3118 hybrid_clock.cc:648] HybridClock initialized: now 1790570591159810 us; error 40 us; skew 500 ppm
Sep 28 04:43:11 dist-test-slave-j5c3 krb5kdc[3001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570591, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:11.162664 3118 init.cc:377] Logged in from keytab as kudu/127.1.226.193@KRBTEST.COM (short username kudu)
I20260928 04:43:11.163211 3118 webserver.cc:460] Webserver started at http://127.1.226.193:42009/ using document root <none> and password file <none>
I20260928 04:43:11.163483 3118 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:11.163561 3118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:11.163739 3118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:11.165047 3118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-0/data/instance:
uuid: "780b2a558b5644c4b7e64975f9293906"
format_stamp: "Formatted at 2026-09-28 04:43:11 on dist-test-slave-j5c3"
server_key: "491bf0f570cc1d629e765cde908e446b"
server_key_iv: "40a3c7bb021f9d7597a4df5b3e79ecca"
server_key_version: "encryptionkey@0"
I20260928 04:43:11.165467 3118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance:
uuid: "780b2a558b5644c4b7e64975f9293906"
format_stamp: "Formatted at 2026-09-28 04:43:11 on dist-test-slave-j5c3"
server_key: "491bf0f570cc1d629e765cde908e446b"
server_key_iv: "40a3c7bb021f9d7597a4df5b3e79ecca"
server_key_version: "encryptionkey@0"
I20260928 04:43:11.166978 3118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:43:11.167840 3133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:11.168075 3118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:11.168210 3118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "780b2a558b5644c4b7e64975f9293906"
format_stamp: "Formatted at 2026-09-28 04:43:11 on dist-test-slave-j5c3"
server_key: "491bf0f570cc1d629e765cde908e446b"
server_key_iv: "40a3c7bb021f9d7597a4df5b3e79ecca"
server_key_version: "encryptionkey@0"
I20260928 04:43:11.168285 3118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:11.181955 3118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:11.184248 3118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:11.184397 3118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:11.184929 3118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:11.184994 3118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:11.185031 3118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:11.185062 3118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 04:43:11 dist-test-slave-j5c3 krb5kdc[3001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570591, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:11.191618 3118 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:37671
I20260928 04:43:11.191694 3248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:37671 every 8 connection(s)
I20260928 04:43:11.192041 3118 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
I20260928 04:43:11.192654 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 3118
I20260928 04:43:11.192730 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance
I20260928 04:43:11.192935 1931 external_mini_cluster.cc:1428] Setting key 6331dadf5ae63748b45c76f4baa46e41
I20260928 04:43:11.194988 3112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.185373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:40835 (local address 127.1.226.254:45591)
0928 04:43:11.185417 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:11.185419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:11.185637 (+ 218us) server_negotiation.cc:407] Connection header received
0928 04:43:11.186156 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:11.186157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:11.186207 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:11.186264 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:11.187174 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.188041 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.189208 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.189557 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.189629 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.191928 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:11.191930 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:11.191930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:11.191937 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:11.193640 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:11.194029 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:11.194030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:11.194031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:11.194090 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:11.194341 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:11.194342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:11.194342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:11.194460 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:11.194534 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:11.194806 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 04:43:11.194895 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:43:11.195174 3142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.185383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:45591 (local address 127.1.226.193:40835)
0928 04:43:11.185522 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:11.185556 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:11.186051 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:11.186340 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:11.186344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:11.186733 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:11.187042 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.187049 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.188140 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.188141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.189104 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.189107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.189662 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.189663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.189792 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.190357 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:11.190366 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:11.191840 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:11.193731 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:11.193737 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:11.193739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:11.193957 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:11.194160 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:11.194161 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:11.194161 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:11.194276 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:11.194524 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:11.194530 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:11.194676 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 04:43:11.194791 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 04:43:11.194955 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":81,"threads_started":1}
I20260928 04:43:11.195605 3249 heartbeater.cc:347] Connected to a master server at 127.1.226.254:45591
I20260928 04:43:11.195688 3249 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:11.195876 3249 heartbeater.cc:511] Master 127.1.226.254:45591 requested a full tablet report, sending...
I20260928 04:43:11.196465 3046 ts_manager.cc:194] Registered new tserver with Master: 780b2a558b5644c4b7e64975f9293906 (127.1.226.193:37671)
I20260928 04:43:11.197453 3046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.193@KRBTEST.COM'} at 127.1.226.193:40835
WARNING: no policy specified for kudu/127.1.226.194@KRBTEST.COM; defaulting to no policy
I20260928 04:43:11.203433 3112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.197675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:33943 (local address 127.1.226.254:45591)
0928 04:43:11.197715 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:11.197717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:11.197725 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:43:11.197745 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:11.197746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:11.197755 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:11.197811 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:11.198174 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.199000 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.199687 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.200000 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.200082 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.201043 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:11.201044 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:11.201045 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:11.201052 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:11.202403 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:11.202819 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:11.202820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:11.202820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:11.202868 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:11.203019 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:11.203019 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:11.203020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:11.203106 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:11.203149 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:11.203280 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 04:43:11.203373 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:43:11.203459 3256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.196761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:45591 (local address 127.1.226.193:33943)
0928 04:43:11.196899 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:11.196909 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:11.196956 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:11.197831 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:11.197832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:11.198036 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:11.198091 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.198093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.199111 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.199112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.199606 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.199608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.200094 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.200095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.200218 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.200670 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:11.200674 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:11.200956 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:11.202490 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:11.202491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:11.202492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:11.202749 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:11.202921 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:11.202922 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:11.202922 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:11.202975 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:11.203149 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:11.203150 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:11.203207 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 04:43:11.203258 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 04:43:11.203335 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
I20260928 04:43:11.218936 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:0
--local_ip_for_outbound_sockets=127.1.226.194
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:45591
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:44349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:11.305809 3258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:11.306032 3258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:11.306056 3258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:11.307246 3258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:11.307291 3258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:11.307312 3258 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 04:43:11.307354 3258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:43:11.308795 3258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:44349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.226.254:45591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:11.309026 3258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:11.309268 3258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:11.314553 3266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:11.314560 3258 server_base.cc:1034] running on GCE node
W20260928 04:43:11.314531 3264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:11.314532 3263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:11.314957 3258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:11.315232 3258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:11.316388 3258 hybrid_clock.cc:648] HybridClock initialized: now 1790570591316384 us; error 41 us; skew 500 ppm
Sep 28 04:43:11 dist-test-slave-j5c3 krb5kdc[3001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570591, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:11.319199 3258 init.cc:377] Logged in from keytab as kudu/127.1.226.194@KRBTEST.COM (short username kudu)
I20260928 04:43:11.319655 3258 webserver.cc:460] Webserver started at http://127.1.226.194:39317/ using document root <none> and password file <none>
I20260928 04:43:11.319926 3258 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:11.319981 3258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:11.320210 3258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:11.321417 3258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-1/data/instance:
uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a"
format_stamp: "Formatted at 2026-09-28 04:43:11 on dist-test-slave-j5c3"
server_key: "2f72f54b934e4af2d017ac2f15c39093"
server_key_iv: "7c7f747d5afcc67f906864286a409a95"
server_key_version: "encryptionkey@0"
I20260928 04:43:11.321854 3258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance:
uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a"
format_stamp: "Formatted at 2026-09-28 04:43:11 on dist-test-slave-j5c3"
server_key: "2f72f54b934e4af2d017ac2f15c39093"
server_key_iv: "7c7f747d5afcc67f906864286a409a95"
server_key_version: "encryptionkey@0"
I20260928 04:43:11.323452 3258 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:43:11.324486 3273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:11.324666 3258 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:43:11.324769 3258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a"
format_stamp: "Formatted at 2026-09-28 04:43:11 on dist-test-slave-j5c3"
server_key: "2f72f54b934e4af2d017ac2f15c39093"
server_key_iv: "7c7f747d5afcc67f906864286a409a95"
server_key_version: "encryptionkey@0"
I20260928 04:43:11.324841 3258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:11.339744 3258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:11.341914 3258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:11.342064 3258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:11.342623 3258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:11.342664 3258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:11.342689 3258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:11.342732 3258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:11.349615 3258 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:36237
I20260928 04:43:11.349709 3388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:36237 every 8 connection(s)
I20260928 04:43:11.350025 3258 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
Sep 28 04:43:11 dist-test-slave-j5c3 krb5kdc[3001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570591, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:11.353523 3112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.343224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:60597 (local address 127.1.226.254:45591)
0928 04:43:11.343277 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:11.343279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:11.343506 (+ 227us) server_negotiation.cc:407] Connection header received
0928 04:43:11.344091 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:11.344094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:11.344108 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:11.344249 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:11.344847 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.345698 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.347268 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.347604 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.347677 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.350647 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:11.350649 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:11.350650 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:11.350659 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:11.352327 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:11.352751 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:11.352752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:11.352752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:11.352813 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:11.353025 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:11.353026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:11.353026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:11.353126 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:11.353188 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:11.353399 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 04:43:11.353459 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 04:43:11.353699 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.343213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:45591 (local address 127.1.226.194:60597)
0928 04:43:11.343390 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:11.343426 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:11.343945 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:11.344332 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:11.344335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:11.344573 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:11.344763 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.344767 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.345802 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.345802 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.347189 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.347192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.347703 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.347705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.347873 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.348607 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:11.348620 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:11.350521 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:11.352415 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:11.352422 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:11.352423 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:11.352673 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:11.352873 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:11.352873 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:11.352873 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:11.352973 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:11.353182 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:11.353186 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:11.353298 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 04:43:11.353376 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 04:43:11.353499 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":105,"threads_started":1}
I20260928 04:43:11.354312 3389 heartbeater.cc:347] Connected to a master server at 127.1.226.254:45591
I20260928 04:43:11.354405 3389 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:11.354597 3389 heartbeater.cc:511] Master 127.1.226.254:45591 requested a full tablet report, sending...
I20260928 04:43:11.355118 3045 ts_manager.cc:194] Registered new tserver with Master: 2e5ffb4c14d347b6abbddd50bc38fb7a (127.1.226.194:36237)
I20260928 04:43:11.355712 3045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.194@KRBTEST.COM'} at 127.1.226.194:60597
I20260928 04:43:11.356848 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 3258
I20260928 04:43:11.356947 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance
I20260928 04:43:11.357141 1931 external_mini_cluster.cc:1428] Setting key 0558df61b96460d8fa3d86053fe9bab9
I20260928 04:43:11.361752 3112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.355563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:41863 (local address 127.1.226.254:45591)
0928 04:43:11.355609 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:11.355611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:11.355720 (+ 109us) server_negotiation.cc:407] Connection header received
0928 04:43:11.355780 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:11.355781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:11.355791 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:11.355840 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:11.356427 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.356971 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.357766 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.357965 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.358045 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.359033 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:11.359034 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:11.359035 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:11.359044 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:11.360530 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:11.360947 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:11.360948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:11.360948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:11.361018 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:11.361264 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:11.361266 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:11.361266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:11.361408 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:11.361469 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:11.361631 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 04:43:11.361689 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:43:11.361852 3395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.355446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:45591 (local address 127.1.226.194:41863)
0928 04:43:11.355642 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:11.355653 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:11.355722 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:11.355886 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:11.355887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:11.356099 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:11.356341 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.356343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.357067 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.357068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.357664 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.357667 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.358045 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.358046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.358125 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.358605 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:11.358610 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:11.358908 (+ 298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:11.360611 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:11.360612 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:11.360612 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:11.360846 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:11.361089 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:11.361090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:11.361091 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:11.361166 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:11.361479 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:11.361480 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:11.361532 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 04:43:11.361611 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 04:43:11.361728 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.1.226.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
I20260928 04:43:11.382920 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:0
--local_ip_for_outbound_sockets=127.1.226.195
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:45591
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:44349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:11.472574 3398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:11.472846 3398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:11.472889 3398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:11.474220 3398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:11.474287 3398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:11.474329 3398 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 04:43:11.474385 3398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:43:11.476002 3398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:44349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.226.254:45591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:11.476339 3398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:11.476608 3398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:11.482146 3406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:11.482167 3403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:11.482151 3404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:11.482260 3398 server_base.cc:1034] running on GCE node
I20260928 04:43:11.482597 3398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:11.482900 3398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:11.484051 3398 hybrid_clock.cc:648] HybridClock initialized: now 1790570591484031 us; error 35 us; skew 500 ppm
Sep 28 04:43:11 dist-test-slave-j5c3 krb5kdc[3001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570591, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:11.486809 3398 init.cc:377] Logged in from keytab as kudu/127.1.226.195@KRBTEST.COM (short username kudu)
I20260928 04:43:11.487283 3398 webserver.cc:460] Webserver started at http://127.1.226.195:41975/ using document root <none> and password file <none>
I20260928 04:43:11.487507 3398 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:11.487551 3398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:11.487725 3398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:11.488915 3398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-2/data/instance:
uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe"
format_stamp: "Formatted at 2026-09-28 04:43:11 on dist-test-slave-j5c3"
server_key: "909c7d8d777f2a45d1951b99148b86ba"
server_key_iv: "84fd4a3a1539958cf2182d3767215999"
server_key_version: "encryptionkey@0"
I20260928 04:43:11.489378 3398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance:
uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe"
format_stamp: "Formatted at 2026-09-28 04:43:11 on dist-test-slave-j5c3"
server_key: "909c7d8d777f2a45d1951b99148b86ba"
server_key_iv: "84fd4a3a1539958cf2182d3767215999"
server_key_version: "encryptionkey@0"
I20260928 04:43:11.490893 3398 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:43:11.491761 3413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:11.491995 3398 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:11.492091 3398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe"
format_stamp: "Formatted at 2026-09-28 04:43:11 on dist-test-slave-j5c3"
server_key: "909c7d8d777f2a45d1951b99148b86ba"
server_key_iv: "84fd4a3a1539958cf2182d3767215999"
server_key_version: "encryptionkey@0"
I20260928 04:43:11.492201 3398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:11.502019 3398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:11.504259 3398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:11.504405 3398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:11.504930 3398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:11.504969 3398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:11.504992 3398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:11.505033 3398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 04:43:11 dist-test-slave-j5c3 krb5kdc[3001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570591, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:11.512012 3398 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:39555
I20260928 04:43:11.512253 3528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:39555 every 8 connection(s)
I20260928 04:43:11.512486 3398 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
I20260928 04:43:11.514215 3112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.505287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:60717 (local address 127.1.226.254:45591)
0928 04:43:11.505332 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:11.505334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:11.505572 (+ 238us) server_negotiation.cc:407] Connection header received
0928 04:43:11.506076 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:11.506077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:11.506091 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:11.506155 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:11.506801 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.507513 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.508638 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.508843 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.508922 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.511541 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:11.511542 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:11.511542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:11.511549 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:11.513161 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:11.513495 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:11.513496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:11.513496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:11.513547 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:11.513726 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:11.513727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:11.513727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:11.513838 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:11.513885 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:11.514099 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 04:43:11.514154 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 04:43:11.514446 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.505294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:45591 (local address 127.1.226.195:60717)
0928 04:43:11.505460 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:11.505493 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:11.505977 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:11.506230 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:11.506233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:11.506527 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:11.506701 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.506705 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.507629 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.507630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.508546 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.508548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.508942 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.508943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.509048 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.509577 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:11.509588 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:11.511458 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:11.513227 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:11.513231 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:11.513232 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:11.513430 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:11.513595 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:11.513595 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:11.513595 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:11.513674 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:11.513890 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:11.513894 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:11.513999 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 04:43:11.514080 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 04:43:11.514234 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":102,"threads_started":1}
I20260928 04:43:11.514878 3529 heartbeater.cc:347] Connected to a master server at 127.1.226.254:45591
I20260928 04:43:11.514954 3529 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:11.515136 3529 heartbeater.cc:511] Master 127.1.226.254:45591 requested a full tablet report, sending...
I20260928 04:43:11.515512 3046 ts_manager.cc:194] Registered new tserver with Master: efa371b22cc8498bb8ec4cbbf76fa5fe (127.1.226.195:39555)
I20260928 04:43:11.516021 3046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.195@KRBTEST.COM'} at 127.1.226.195:60717
I20260928 04:43:11.517925 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 3398
I20260928 04:43:11.518028 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance
I20260928 04:43:11.518265 1931 external_mini_cluster.cc:1428] Setting key bab657a75d55006ffbbf31b33ea1ac90
I20260928 04:43:11.519209 1931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:43:11.522289 3112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.515948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:59473 (local address 127.1.226.254:45591)
0928 04:43:11.515985 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:11.515987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:11.516091 (+ 104us) server_negotiation.cc:407] Connection header received
0928 04:43:11.516195 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:11.516196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:11.516207 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:11.516263 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:11.516577 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.517176 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.518219 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.518461 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.518532 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.519506 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:11.519507 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:11.519508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:11.519514 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:11.521053 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:11.521553 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:11.521554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:11.521555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:11.521603 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:11.521839 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:11.521839 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:11.521839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:11.521929 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:11.521977 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:11.522181 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 04:43:11.522232 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 04:43:11.522440 3535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.515903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:45591 (local address 127.1.226.195:59473)
0928 04:43:11.516019 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:11.516029 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:11.516106 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:11.516274 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:11.516275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:11.516456 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:11.516502 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.516503 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.517288 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.517289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.518134 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.518136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.518546 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.518547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.518611 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.519098 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:11.519103 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:11.519433 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:11.521156 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:11.521157 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:11.521158 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:11.521450 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:11.521672 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:11.521673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:11.521674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:11.521763 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:11.521992 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:11.521993 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:11.522051 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 04:43:11.522170 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 04:43:11.522294 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":59,"threads_started":1}
Sep 28 04:43:11 dist-test-slave-j5c3 krb5kdc[3001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570591, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 04:43:11 dist-test-slave-j5c3 krb5kdc[3001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570591, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:11.554507 3112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.546797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51378 (local address 127.1.226.254:45591)
0928 04:43:11.546869 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:11.546871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:11.546881 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:43:11.546908 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:11.546909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:11.546919 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:11.547015 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:11.547623 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.548362 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.549115 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.549323 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.549407 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.551848 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:11.551849 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:11.551850 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:11.551857 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:11.553246 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:11.553601 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:11.553602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:11.553602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:11.553651 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:11.553871 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:11.553872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:11.553872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:11.554012 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:11.554085 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:11.554284 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 04:43:11.554349 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 04:43:11.556473 3046 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:43:11.558146 3046 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:43:11.564986 3547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.560606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:39555 (local address 127.0.0.1:40160)
0928 04:43:11.560970 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:11.560979 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:11.561016 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:11.561419 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:11.561420 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:11.561421 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:11.561566 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.561568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.562449 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.562450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.563524 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.563527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.564641 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.564642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.564720 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.564736 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 04:43:11.564824 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 04:43:11.564875 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":106,"thread_start_us":97,"threads_started":1}
I20260928 04:43:11.565038 3549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.560862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40160 (local address 127.1.226.195:39555)
0928 04:43:11.561213 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:11.561217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:11.561225 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:43:11.561255 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:11.561256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:11.561268 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:11.561359 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:11.561644 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.562363 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.563735 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.564526 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.564643 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.564712 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:11.564844 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 04:43:11.564891 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":83,"threads_started":1}
I20260928 04:43:11.565289 3546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.560593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:37671 (local address 127.0.0.1:37294)
0928 04:43:11.560867 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:11.560879 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:11.560944 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:11.561423 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:11.561424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:11.561427 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:11.561498 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.561500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.563023 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.563024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.563978 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.563980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.565044 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.565046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.565139 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.565144 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:43:11.565189 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 04:43:11.565233 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":113,"threads_started":1}
I20260928 04:43:11.565711 3551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.561046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37294 (local address 127.1.226.193:37671)
0928 04:43:11.561227 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:11.561232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:11.561242 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:43:11.561285 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:11.561287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:11.561305 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:11.561591 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:11.561659 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.562916 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.564059 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.564921 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.565435 (+ 514us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.565519 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:11.565544 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 04:43:11.565579 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":79,"threads_started":1}
I20260928 04:43:11.566154 3454 tablet_service.cc:1467] Processing CreateTablet for tablet 21928e2a3879497992842f2d2705780a (DEFAULT_TABLE table=test-table [id=eb2f248f41674dde97332f879a8c2036]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:11.566499 3454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21928e2a3879497992842f2d2705780a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:11.566402 3183 tablet_service.cc:1467] Processing CreateTablet for tablet 21928e2a3879497992842f2d2705780a (DEFAULT_TABLE table=test-table [id=eb2f248f41674dde97332f879a8c2036]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:11.566740 3183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21928e2a3879497992842f2d2705780a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:11.569415 3548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.560529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:36237 (local address 127.0.0.1:54168)
0928 04:43:11.560749 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:11.560779 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:11.560863 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:11.561346 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:11.561350 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:11.561352 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:11.561430 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.561431 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.564049 (+ 2618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.564050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.564890 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.564892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.569088 (+ 4196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.569090 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.569196 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.569202 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 04:43:11.569267 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 04:43:11.569321 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":56,"threads_started":1}
I20260928 04:43:11.569468 3550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.560906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54168 (local address 127.1.226.194:36237)
0928 04:43:11.561119 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:11.561142 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:11.561157 (+ 15us) server_negotiation.cc:407] Connection header received
0928 04:43:11.561207 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:11.561210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:11.561230 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:11.561486 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:11.561559 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.562911 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.564982 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.568975 (+ 3993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.569083 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.569166 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:11.569286 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 04:43:11.569349 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":96,"threads_started":1}
I20260928 04:43:11.569741 3552 tablet_bootstrap.cc:492] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe: Bootstrap starting.
I20260928 04:43:11.570286 3315 tablet_service.cc:1467] Processing CreateTablet for tablet 21928e2a3879497992842f2d2705780a (DEFAULT_TABLE table=test-table [id=eb2f248f41674dde97332f879a8c2036]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:11.570588 3315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21928e2a3879497992842f2d2705780a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:11.570734 3552 tablet_bootstrap.cc:654] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:11.571023 3552 log.cc:824] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:11.571380 3553 tablet_bootstrap.cc:492] T 21928e2a3879497992842f2d2705780a P 780b2a558b5644c4b7e64975f9293906: Bootstrap starting.
I20260928 04:43:11.571954 3552 tablet_bootstrap.cc:492] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe: No bootstrap required, opened a new log
I20260928 04:43:11.572016 3552 ts_tablet_manager.cc:1397] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:43:11.572022 3553 tablet_bootstrap.cc:654] T 21928e2a3879497992842f2d2705780a P 780b2a558b5644c4b7e64975f9293906: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:11.572441 3553 log.cc:824] T 21928e2a3879497992842f2d2705780a P 780b2a558b5644c4b7e64975f9293906: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:11.574190 3552 raft_consensus.cc:348] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } } peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } }
I20260928 04:43:11.574381 3552 raft_consensus.cc:374] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:11.574427 3552 raft_consensus.cc:729] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa371b22cc8498bb8ec4cbbf76fa5fe, State: Initialized, Role: FOLLOWER
I20260928 04:43:11.574473 3556 tablet_bootstrap.cc:492] T 21928e2a3879497992842f2d2705780a P 2e5ffb4c14d347b6abbddd50bc38fb7a: Bootstrap starting.
I20260928 04:43:11.574609 3552 consensus_queue.cc:260] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } } peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } }
I20260928 04:43:11.574901 3552 ts_tablet_manager.cc:1428] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:43:11.575045 3529 heartbeater.cc:503] Master 127.1.226.254:45591 was elected leader, sending a full tablet report...
I20260928 04:43:11.575125 3556 tablet_bootstrap.cc:654] T 21928e2a3879497992842f2d2705780a P 2e5ffb4c14d347b6abbddd50bc38fb7a: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:11.575466 3556 log.cc:824] T 21928e2a3879497992842f2d2705780a P 2e5ffb4c14d347b6abbddd50bc38fb7a: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:11.575611 3553 tablet_bootstrap.cc:492] T 21928e2a3879497992842f2d2705780a P 780b2a558b5644c4b7e64975f9293906: No bootstrap required, opened a new log
I20260928 04:43:11.575685 3553 ts_tablet_manager.cc:1397] T 21928e2a3879497992842f2d2705780a P 780b2a558b5644c4b7e64975f9293906: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260928 04:43:11.576259 3556 tablet_bootstrap.cc:492] T 21928e2a3879497992842f2d2705780a P 2e5ffb4c14d347b6abbddd50bc38fb7a: No bootstrap required, opened a new log
I20260928 04:43:11.576335 3556 ts_tablet_manager.cc:1397] T 21928e2a3879497992842f2d2705780a P 2e5ffb4c14d347b6abbddd50bc38fb7a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:43:11.577775 3553 raft_consensus.cc:348] T 21928e2a3879497992842f2d2705780a P 780b2a558b5644c4b7e64975f9293906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } } peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } }
I20260928 04:43:11.577960 3553 raft_consensus.cc:374] T 21928e2a3879497992842f2d2705780a P 780b2a558b5644c4b7e64975f9293906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:11.577996 3553 raft_consensus.cc:729] T 21928e2a3879497992842f2d2705780a P 780b2a558b5644c4b7e64975f9293906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 780b2a558b5644c4b7e64975f9293906, State: Initialized, Role: FOLLOWER
I20260928 04:43:11.578120 3553 consensus_queue.cc:260] T 21928e2a3879497992842f2d2705780a P 780b2a558b5644c4b7e64975f9293906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } } peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } }
I20260928 04:43:11.578783 3553 ts_tablet_manager.cc:1428] T 21928e2a3879497992842f2d2705780a P 780b2a558b5644c4b7e64975f9293906: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 04:43:11.578783 3556 raft_consensus.cc:348] T 21928e2a3879497992842f2d2705780a P 2e5ffb4c14d347b6abbddd50bc38fb7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } } peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } }
I20260928 04:43:11.578945 3556 raft_consensus.cc:374] T 21928e2a3879497992842f2d2705780a P 2e5ffb4c14d347b6abbddd50bc38fb7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:11.578943 3249 heartbeater.cc:503] Master 127.1.226.254:45591 was elected leader, sending a full tablet report...
I20260928 04:43:11.578992 3556 raft_consensus.cc:729] T 21928e2a3879497992842f2d2705780a P 2e5ffb4c14d347b6abbddd50bc38fb7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e5ffb4c14d347b6abbddd50bc38fb7a, State: Initialized, Role: FOLLOWER
I20260928 04:43:11.579103 3556 consensus_queue.cc:260] T 21928e2a3879497992842f2d2705780a P 2e5ffb4c14d347b6abbddd50bc38fb7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } } peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } }
I20260928 04:43:11.579380 3556 ts_tablet_manager.cc:1428] T 21928e2a3879497992842f2d2705780a P 2e5ffb4c14d347b6abbddd50bc38fb7a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:43:11.579510 3389 heartbeater.cc:503] Master 127.1.226.254:45591 was elected leader, sending a full tablet report...
W20260928 04:43:11.600586 3390 tablet.cc:2369] T 21928e2a3879497992842f2d2705780a P 2e5ffb4c14d347b6abbddd50bc38fb7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:43:11.692811 3250 tablet.cc:2369] T 21928e2a3879497992842f2d2705780a P 780b2a558b5644c4b7e64975f9293906: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:43:11.763082 3530 tablet.cc:2369] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:43:11.774204 3557 raft_consensus.cc:482] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:43:11.774333 3557 raft_consensus.cc:504] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } } peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } }
I20260928 04:43:11.774631 3557 leader_election.cc:288] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e5ffb4c14d347b6abbddd50bc38fb7a (127.1.226.194:36237), 780b2a558b5644c4b7e64975f9293906 (127.1.226.193:37671)
I20260928 04:43:11.777578 3551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.774979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:43885 (local address 127.1.226.193:37671)
0928 04:43:11.775034 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:11.775037 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:11.775047 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:43:11.775112 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:11.775113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:11.775122 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:11.775198 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:11.775350 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.775835 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.776828 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.777204 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.777301 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.777311 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:11.777470 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 04:43:11.777513 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 04:43:11.777578 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.774874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:37671 (local address 127.1.226.195:43885)
0928 04:43:11.774965 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:11.774980 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:11.775048 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:11.775204 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:11.775205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:11.775207 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:11.775278 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.775280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.775915 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.775916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.776736 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.776739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.777302 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.777303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.777386 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.777389 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:43:11.777460 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 04:43:11.777502 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 04:43:11.777709 3550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.774924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:47195 (local address 127.1.226.194:36237)
0928 04:43:11.774984 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:11.774987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:11.775233 (+ 246us) server_negotiation.cc:407] Connection header received
0928 04:43:11.775303 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:11.775305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:11.775313 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:11.775381 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:11.775550 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.776031 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.776989 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.777363 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.777443 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.777450 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:11.777610 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 04:43:11.777647 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 04:43:11.777738 3561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.774874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:36237 (local address 127.1.226.195:47195)
0928 04:43:11.775129 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:11.775139 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:11.775244 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:11.775383 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:11.775384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:11.775399 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:11.775479 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.775481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.776111 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.776112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.776878 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.776880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.777442 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.777442 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.777525 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.777529 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:43:11.777601 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 04:43:11.777640 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":92,"thread_start_us":110,"threads_started":1}
I20260928 04:43:11.777951 3203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21928e2a3879497992842f2d2705780a" candidate_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "780b2a558b5644c4b7e64975f9293906" is_pre_election: true
I20260928 04:43:11.778028 3343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21928e2a3879497992842f2d2705780a" candidate_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" is_pre_election: true
I20260928 04:43:11.778126 3203 raft_consensus.cc:2454] T 21928e2a3879497992842f2d2705780a P 780b2a558b5644c4b7e64975f9293906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efa371b22cc8498bb8ec4cbbf76fa5fe in term 0.
I20260928 04:43:11.778172 3343 raft_consensus.cc:2454] T 21928e2a3879497992842f2d2705780a P 2e5ffb4c14d347b6abbddd50bc38fb7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efa371b22cc8498bb8ec4cbbf76fa5fe in term 0.
I20260928 04:43:11.778383 3417 leader_election.cc:302] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 780b2a558b5644c4b7e64975f9293906, efa371b22cc8498bb8ec4cbbf76fa5fe; no voters:
I20260928 04:43:11.778502 3557 raft_consensus.cc:2781] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:43:11.778561 3557 raft_consensus.cc:482] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:43:11.778589 3557 raft_consensus.cc:3037] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:11.779685 3557 raft_consensus.cc:504] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } } peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } }
I20260928 04:43:11.779835 3557 leader_election.cc:288] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe [CANDIDATE]: Term 1 election: Requested vote from peers 2e5ffb4c14d347b6abbddd50bc38fb7a (127.1.226.194:36237), 780b2a558b5644c4b7e64975f9293906 (127.1.226.193:37671)
I20260928 04:43:11.779955 3343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21928e2a3879497992842f2d2705780a" candidate_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a"
I20260928 04:43:11.779975 3203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21928e2a3879497992842f2d2705780a" candidate_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "780b2a558b5644c4b7e64975f9293906"
I20260928 04:43:11.780035 3343 raft_consensus.cc:3037] T 21928e2a3879497992842f2d2705780a P 2e5ffb4c14d347b6abbddd50bc38fb7a [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:11.780045 3203 raft_consensus.cc:3037] T 21928e2a3879497992842f2d2705780a P 780b2a558b5644c4b7e64975f9293906 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:11.780900 3343 raft_consensus.cc:2454] T 21928e2a3879497992842f2d2705780a P 2e5ffb4c14d347b6abbddd50bc38fb7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efa371b22cc8498bb8ec4cbbf76fa5fe in term 1.
I20260928 04:43:11.780900 3203 raft_consensus.cc:2454] T 21928e2a3879497992842f2d2705780a P 780b2a558b5644c4b7e64975f9293906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efa371b22cc8498bb8ec4cbbf76fa5fe in term 1.
I20260928 04:43:11.781071 3417 leader_election.cc:302] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 780b2a558b5644c4b7e64975f9293906, efa371b22cc8498bb8ec4cbbf76fa5fe; no voters:
I20260928 04:43:11.781162 3557 raft_consensus.cc:2781] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:11.781294 3557 raft_consensus.cc:686] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe [term 1 LEADER]: Becoming Leader. State: Replica: efa371b22cc8498bb8ec4cbbf76fa5fe, State: Running, Role: LEADER
I20260928 04:43:11.781394 3557 consensus_queue.cc:237] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } } peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } }
I20260928 04:43:11.782267 3046 catalog_manager.cc:5495] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe reported cstate change: term changed from 0 to 1, leader changed from <none> to efa371b22cc8498bb8ec4cbbf76fa5fe (127.1.226.195). New cstate: current_term: 1 leader_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } health_report { overall_health: UNKNOWN } } }
I20260928 04:43:11.791483 3549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.789120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40174 (local address 127.1.226.195:39555)
0928 04:43:11.789174 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:11.789176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:11.789231 (+ 55us) server_negotiation.cc:407] Connection header received
0928 04:43:11.789279 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:11.789280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:11.789295 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:11.789377 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:43:11.789552 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.790032 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.790770 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.790951 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.791005 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.791178 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:43:11.791281 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:43:11.791340 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:11.791380 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 04:43:11.791399 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 04:43:11.793315 3203 raft_consensus.cc:1261] T 21928e2a3879497992842f2d2705780a P 780b2a558b5644c4b7e64975f9293906 [term 1 FOLLOWER]: Refusing update from remote peer efa371b22cc8498bb8ec4cbbf76fa5fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:11.793352 3343 raft_consensus.cc:1261] T 21928e2a3879497992842f2d2705780a P 2e5ffb4c14d347b6abbddd50bc38fb7a [term 1 FOLLOWER]: Refusing update from remote peer efa371b22cc8498bb8ec4cbbf76fa5fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:11.793684 3557 consensus_queue.cc:1035] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:11.793870 3562 consensus_queue.cc:1035] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:11.796077 3565 mvcc.cc:205] Tried to move back new op lower bound from 7334177143982182400 to 7334177143937277952. Current Snapshot: MvccSnapshot[applied={T|T < 7334177143982182400}]
I20260928 04:43:11.796617 3566 mvcc.cc:205] Tried to move back new op lower bound from 7334177143982182400 to 7334177143937277952. Current Snapshot: MvccSnapshot[applied={T|T < 7334177143982182400 or (T in {7334177143982182400})}]
I20260928 04:43:11.796552 3567 mvcc.cc:205] Tried to move back new op lower bound from 7334177143982182400 to 7334177143937277952. Current Snapshot: MvccSnapshot[applied={T|T < 7334177143982182400}]
I20260928 04:43:11.800215 3045 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:43:11.800323 3045 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:43:11.801151 3045 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2bec407195a84052afacd9c2624e3bc0: Sending DeleteTablet for 3 replicas of tablet 21928e2a3879497992842f2d2705780a
I20260928 04:43:11.801507 3315 tablet_service.cc:1514] Processing DeleteTablet for tablet 21928e2a3879497992842f2d2705780a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:11 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:54168
I20260928 04:43:11.801510 3183 tablet_service.cc:1514] Processing DeleteTablet for tablet 21928e2a3879497992842f2d2705780a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:11 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:37294
I20260928 04:43:11.801508 3454 tablet_service.cc:1514] Processing DeleteTablet for tablet 21928e2a3879497992842f2d2705780a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:11 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:40160
I20260928 04:43:11.801720 3576 tablet_replica.cc:330] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe: stopping tablet replica
I20260928 04:43:11.801836 3577 tablet_replica.cc:330] T 21928e2a3879497992842f2d2705780a P 780b2a558b5644c4b7e64975f9293906: stopping tablet replica
I20260928 04:43:11.801937 3577 raft_consensus.cc:2229] T 21928e2a3879497992842f2d2705780a P 780b2a558b5644c4b7e64975f9293906 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:43:11.802073 3576 raft_consensus.cc:2229] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:43:11.802110 3045 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 04:43:11.802212 3576 raft_consensus.cc:2258] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 04:43:11.802260 3045 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:43:11.802399 3577 raft_consensus.cc:2258] T 21928e2a3879497992842f2d2705780a P 780b2a558b5644c4b7e64975f9293906 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:11.803018 3577 ts_tablet_manager.cc:1905] T 21928e2a3879497992842f2d2705780a P 780b2a558b5644c4b7e64975f9293906: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:11.803503 3454 tablet_service.cc:1467] Processing CreateTablet for tablet f9e04284c5de43a293042d6f221e05dc (DEFAULT_TABLE table=test-table [id=c16b589b55b14ebaa7830b75a8a8aaef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:11.803586 3183 tablet_service.cc:1467] Processing CreateTablet for tablet f9e04284c5de43a293042d6f221e05dc (DEFAULT_TABLE table=test-table [id=c16b589b55b14ebaa7830b75a8a8aaef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:11.803623 3454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9e04284c5de43a293042d6f221e05dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:11.803727 3183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9e04284c5de43a293042d6f221e05dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:11.805006 3553 tablet_bootstrap.cc:492] T f9e04284c5de43a293042d6f221e05dc P 780b2a558b5644c4b7e64975f9293906: Bootstrap starting.
I20260928 04:43:11.805073 3315 tablet_service.cc:1467] Processing CreateTablet for tablet f9e04284c5de43a293042d6f221e05dc (DEFAULT_TABLE table=test-table [id=c16b589b55b14ebaa7830b75a8a8aaef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:11.805177 3315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9e04284c5de43a293042d6f221e05dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:11.805265 3577 ts_tablet_manager.cc:1918] T 21928e2a3879497992842f2d2705780a P 780b2a558b5644c4b7e64975f9293906: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:43:11.805333 3577 log.cc:1196] T 21928e2a3879497992842f2d2705780a P 780b2a558b5644c4b7e64975f9293906: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-0/wal/wals/21928e2a3879497992842f2d2705780a
I20260928 04:43:11.803524 3576 ts_tablet_manager.cc:1905] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:11.805456 3553 tablet_bootstrap.cc:654] T f9e04284c5de43a293042d6f221e05dc P 780b2a558b5644c4b7e64975f9293906: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:11.805595 3577 ts_tablet_manager.cc:1939] T 21928e2a3879497992842f2d2705780a P 780b2a558b5644c4b7e64975f9293906: Deleting consensus metadata
I20260928 04:43:11.806180 3553 tablet_bootstrap.cc:492] T f9e04284c5de43a293042d6f221e05dc P 780b2a558b5644c4b7e64975f9293906: No bootstrap required, opened a new log
I20260928 04:43:11.806234 3553 ts_tablet_manager.cc:1397] T f9e04284c5de43a293042d6f221e05dc P 780b2a558b5644c4b7e64975f9293906: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 04:43:11.806555 3552 tablet_bootstrap.cc:492] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe: Bootstrap starting.
I20260928 04:43:11.806552 3553 raft_consensus.cc:348] T f9e04284c5de43a293042d6f221e05dc P 780b2a558b5644c4b7e64975f9293906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } }
I20260928 04:43:11.806629 3553 raft_consensus.cc:374] T f9e04284c5de43a293042d6f221e05dc P 780b2a558b5644c4b7e64975f9293906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:11.806650 3553 raft_consensus.cc:729] T f9e04284c5de43a293042d6f221e05dc P 780b2a558b5644c4b7e64975f9293906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 780b2a558b5644c4b7e64975f9293906, State: Initialized, Role: FOLLOWER
I20260928 04:43:11.806696 3553 consensus_queue.cc:260] T f9e04284c5de43a293042d6f221e05dc P 780b2a558b5644c4b7e64975f9293906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } }
I20260928 04:43:11.806865 3553 ts_tablet_manager.cc:1428] T f9e04284c5de43a293042d6f221e05dc P 780b2a558b5644c4b7e64975f9293906: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 04:43:11.806887 3576 ts_tablet_manager.cc:1918] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:43:11.806951 3576 log.cc:1196] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-2/wal/wals/21928e2a3879497992842f2d2705780a
I20260928 04:43:11.807180 3576 ts_tablet_manager.cc:1939] T 21928e2a3879497992842f2d2705780a P efa371b22cc8498bb8ec4cbbf76fa5fe: Deleting consensus metadata
I20260928 04:43:11.807412 3552 tablet_bootstrap.cc:654] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:11.807529 3032 catalog_manager.cc:4841] TS efa371b22cc8498bb8ec4cbbf76fa5fe (127.1.226.195:39555): tablet 21928e2a3879497992842f2d2705780a (table test-table [id=eb2f248f41674dde97332f879a8c2036]) successfully deleted
I20260928 04:43:11.807703 3556 tablet_bootstrap.cc:492] T f9e04284c5de43a293042d6f221e05dc P 2e5ffb4c14d347b6abbddd50bc38fb7a: Bootstrap starting.
I20260928 04:43:11.807992 3552 tablet_bootstrap.cc:492] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe: No bootstrap required, opened a new log
I20260928 04:43:11.808023 3552 ts_tablet_manager.cc:1397] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:11.808079 3556 tablet_bootstrap.cc:654] T f9e04284c5de43a293042d6f221e05dc P 2e5ffb4c14d347b6abbddd50bc38fb7a: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:11.808120 3552 raft_consensus.cc:348] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } }
I20260928 04:43:11.808225 3552 raft_consensus.cc:374] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:11.808254 3552 raft_consensus.cc:729] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa371b22cc8498bb8ec4cbbf76fa5fe, State: Initialized, Role: FOLLOWER
I20260928 04:43:11.808298 3552 consensus_queue.cc:260] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } }
I20260928 04:43:11.808386 3552 ts_tablet_manager.cc:1428] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:11.808671 3033 catalog_manager.cc:4841] TS 780b2a558b5644c4b7e64975f9293906 (127.1.226.193:37671): tablet 21928e2a3879497992842f2d2705780a (table test-table [id=eb2f248f41674dde97332f879a8c2036]) successfully deleted
I20260928 04:43:11.809109 3575 tablet_replica.cc:330] T 21928e2a3879497992842f2d2705780a P 2e5ffb4c14d347b6abbddd50bc38fb7a: stopping tablet replica
I20260928 04:43:11.809247 3575 raft_consensus.cc:2229] T 21928e2a3879497992842f2d2705780a P 2e5ffb4c14d347b6abbddd50bc38fb7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:43:11.809309 3556 tablet_bootstrap.cc:492] T f9e04284c5de43a293042d6f221e05dc P 2e5ffb4c14d347b6abbddd50bc38fb7a: No bootstrap required, opened a new log
I20260928 04:43:11.809376 3556 ts_tablet_manager.cc:1397] T f9e04284c5de43a293042d6f221e05dc P 2e5ffb4c14d347b6abbddd50bc38fb7a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 04:43:11.809401 3575 raft_consensus.cc:2258] T 21928e2a3879497992842f2d2705780a P 2e5ffb4c14d347b6abbddd50bc38fb7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:11.809518 3556 raft_consensus.cc:348] T f9e04284c5de43a293042d6f221e05dc P 2e5ffb4c14d347b6abbddd50bc38fb7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } }
I20260928 04:43:11.809588 3556 raft_consensus.cc:374] T f9e04284c5de43a293042d6f221e05dc P 2e5ffb4c14d347b6abbddd50bc38fb7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:11.809608 3556 raft_consensus.cc:729] T f9e04284c5de43a293042d6f221e05dc P 2e5ffb4c14d347b6abbddd50bc38fb7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e5ffb4c14d347b6abbddd50bc38fb7a, State: Initialized, Role: FOLLOWER
I20260928 04:43:11.809656 3556 consensus_queue.cc:260] T f9e04284c5de43a293042d6f221e05dc P 2e5ffb4c14d347b6abbddd50bc38fb7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } }
I20260928 04:43:11.809739 3556 ts_tablet_manager.cc:1428] T f9e04284c5de43a293042d6f221e05dc P 2e5ffb4c14d347b6abbddd50bc38fb7a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 04:43:11.809852 3575 ts_tablet_manager.cc:1905] T 21928e2a3879497992842f2d2705780a P 2e5ffb4c14d347b6abbddd50bc38fb7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:11.811455 3575 ts_tablet_manager.cc:1918] T 21928e2a3879497992842f2d2705780a P 2e5ffb4c14d347b6abbddd50bc38fb7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:43:11.811530 3575 log.cc:1196] T 21928e2a3879497992842f2d2705780a P 2e5ffb4c14d347b6abbddd50bc38fb7a: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-1/wal/wals/21928e2a3879497992842f2d2705780a
I20260928 04:43:11.811811 3575 ts_tablet_manager.cc:1939] T 21928e2a3879497992842f2d2705780a P 2e5ffb4c14d347b6abbddd50bc38fb7a: Deleting consensus metadata
I20260928 04:43:11.812214 3031 catalog_manager.cc:4841] TS 2e5ffb4c14d347b6abbddd50bc38fb7a (127.1.226.194:36237): tablet 21928e2a3879497992842f2d2705780a (table test-table [id=eb2f248f41674dde97332f879a8c2036]) successfully deleted
I20260928 04:43:11.825243 3559 raft_consensus.cc:482] T f9e04284c5de43a293042d6f221e05dc P 780b2a558b5644c4b7e64975f9293906 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:43:11.825345 3559 raft_consensus.cc:504] T f9e04284c5de43a293042d6f221e05dc P 780b2a558b5644c4b7e64975f9293906 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } }
I20260928 04:43:11.825591 3571 raft_consensus.cc:482] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:43:11.825721 3559 leader_election.cc:288] T f9e04284c5de43a293042d6f221e05dc P 780b2a558b5644c4b7e64975f9293906 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e5ffb4c14d347b6abbddd50bc38fb7a (127.1.226.194:36237), efa371b22cc8498bb8ec4cbbf76fa5fe (127.1.226.195:39555)
I20260928 04:43:11.825696 3571 raft_consensus.cc:504] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } }
I20260928 04:43:11.825856 3571 leader_election.cc:288] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 780b2a558b5644c4b7e64975f9293906 (127.1.226.193:37671), 2e5ffb4c14d347b6abbddd50bc38fb7a (127.1.226.194:36237)
I20260928 04:43:11.826551 3203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e04284c5de43a293042d6f221e05dc" candidate_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "780b2a558b5644c4b7e64975f9293906" is_pre_election: true
I20260928 04:43:11.826653 3203 raft_consensus.cc:2454] T f9e04284c5de43a293042d6f221e05dc P 780b2a558b5644c4b7e64975f9293906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efa371b22cc8498bb8ec4cbbf76fa5fe in term 0.
I20260928 04:43:11.826797 3343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e04284c5de43a293042d6f221e05dc" candidate_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" is_pre_election: true
I20260928 04:43:11.826882 3417 leader_election.cc:302] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 780b2a558b5644c4b7e64975f9293906, efa371b22cc8498bb8ec4cbbf76fa5fe; no voters:
I20260928 04:43:11.826937 3343 raft_consensus.cc:2454] T f9e04284c5de43a293042d6f221e05dc P 2e5ffb4c14d347b6abbddd50bc38fb7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efa371b22cc8498bb8ec4cbbf76fa5fe in term 0.
I20260928 04:43:11.826970 3571 raft_consensus.cc:2781] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:43:11.826996 3571 raft_consensus.cc:482] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:43:11.827033 3571 raft_consensus.cc:3037] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:11.827929 3571 raft_consensus.cc:504] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } }
I20260928 04:43:11.828074 3571 leader_election.cc:288] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe [CANDIDATE]: Term 1 election: Requested vote from peers 780b2a558b5644c4b7e64975f9293906 (127.1.226.193:37671), 2e5ffb4c14d347b6abbddd50bc38fb7a (127.1.226.194:36237)
I20260928 04:43:11.828373 3343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e04284c5de43a293042d6f221e05dc" candidate_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a"
I20260928 04:43:11.828446 3343 raft_consensus.cc:3037] T f9e04284c5de43a293042d6f221e05dc P 2e5ffb4c14d347b6abbddd50bc38fb7a [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:11.829205 3343 raft_consensus.cc:2454] T f9e04284c5de43a293042d6f221e05dc P 2e5ffb4c14d347b6abbddd50bc38fb7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efa371b22cc8498bb8ec4cbbf76fa5fe in term 1.
I20260928 04:43:11.829547 3203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e04284c5de43a293042d6f221e05dc" candidate_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "780b2a558b5644c4b7e64975f9293906"
I20260928 04:43:11.829622 3203 raft_consensus.cc:3037] T f9e04284c5de43a293042d6f221e05dc P 780b2a558b5644c4b7e64975f9293906 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:11.830395 3203 raft_consensus.cc:2454] T f9e04284c5de43a293042d6f221e05dc P 780b2a558b5644c4b7e64975f9293906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efa371b22cc8498bb8ec4cbbf76fa5fe in term 1.
I20260928 04:43:11.830502 3550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.826567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:47067 (local address 127.1.226.194:36237)
0928 04:43:11.826646 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:11.826648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:11.826655 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:43:11.826674 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:11.826675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:11.826686 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:11.826737 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:11.827252 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.827917 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.829543 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.830051 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.830101 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.830111 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:11.830376 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 04:43:11.830423 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 04:43:11.830505 3581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.825971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:39555 (local address 127.1.226.193:41963)
0928 04:43:11.826088 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:11.826102 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:11.826155 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:11.826770 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:11.826771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:11.826773 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:11.826854 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.826857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.828048 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.828050 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.829459 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.829463 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.830238 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.830239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.830328 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.830331 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:43:11.830376 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 04:43:11.830416 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":62,"threads_started":1}
I20260928 04:43:11.830693 3549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.826493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:41963 (local address 127.1.226.195:39555)
0928 04:43:11.826564 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:11.826567 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:11.826574 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:43:11.826599 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:11.826600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:11.826610 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:11.826663 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:11.826928 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.827947 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.829586 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.830120 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.830172 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.830179 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:11.830583 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 04:43:11.830629 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 04:43:11.830713 3582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.826165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:36237 (local address 127.1.226.193:47067)
0928 04:43:11.826291 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:11.826303 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:11.826343 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:11.827033 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:11.827033 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:11.827035 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:11.827145 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.827147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.828040 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.828041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.829442 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.829445 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.830208 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.830209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.830283 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.830288 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:43:11.830555 (+ 267us) client_negotiation.cc:240] Negotiation successful
0928 04:43:11.830609 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":103,"thread_start_us":76,"threads_started":1}
I20260928 04:43:11.830992 3417 leader_election.cc:302] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 780b2a558b5644c4b7e64975f9293906, efa371b22cc8498bb8ec4cbbf76fa5fe; no voters:
I20260928 04:43:11.831176 3571 raft_consensus.cc:2781] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:11.831234 3571 raft_consensus.cc:686] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe [term 1 LEADER]: Becoming Leader. State: Replica: efa371b22cc8498bb8ec4cbbf76fa5fe, State: Running, Role: LEADER
I20260928 04:43:11.831207 3483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e04284c5de43a293042d6f221e05dc" candidate_uuid: "780b2a558b5644c4b7e64975f9293906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" is_pre_election: true
I20260928 04:43:11.831283 3571 consensus_queue.cc:237] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } }
I20260928 04:43:11.831610 3343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e04284c5de43a293042d6f221e05dc" candidate_uuid: "780b2a558b5644c4b7e64975f9293906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" is_pre_election: true
I20260928 04:43:11.831698 3343 raft_consensus.cc:2379] T f9e04284c5de43a293042d6f221e05dc P 2e5ffb4c14d347b6abbddd50bc38fb7a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 780b2a558b5644c4b7e64975f9293906 in current term 1: Already voted for candidate efa371b22cc8498bb8ec4cbbf76fa5fe in this term.
I20260928 04:43:11.831781 3045 catalog_manager.cc:5495] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe reported cstate change: term changed from 0 to 1, leader changed from <none> to efa371b22cc8498bb8ec4cbbf76fa5fe (127.1.226.195). New cstate: current_term: 1 leader_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } health_report { overall_health: HEALTHY } } }
I20260928 04:43:11.831905 3135 leader_election.cc:302] T f9e04284c5de43a293042d6f221e05dc P 780b2a558b5644c4b7e64975f9293906 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 780b2a558b5644c4b7e64975f9293906; no voters: 2e5ffb4c14d347b6abbddd50bc38fb7a, efa371b22cc8498bb8ec4cbbf76fa5fe
I20260928 04:43:11.832054 3559 raft_consensus.cc:2726] T f9e04284c5de43a293042d6f221e05dc P 780b2a558b5644c4b7e64975f9293906 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Sep 28 04:43:11 dist-test-slave-j5c3 krb5kdc[3001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570590, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:11.872505 3112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.864976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51390 (local address 127.1.226.254:45591)
0928 04:43:11.865042 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:11.865045 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:11.865169 (+ 124us) server_negotiation.cc:407] Connection header received
0928 04:43:11.865244 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:11.865245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:11.865255 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:11.865322 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:11.865834 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.866381 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.867087 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.867276 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.867354 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.869596 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:11.869597 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:11.869597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:11.869605 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:11.871261 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:11.871678 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:11.871679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:11.871680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:11.871741 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:11.871940 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:11.871941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:11.871941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:11.872048 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:11.872117 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:11.872318 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 04:43:11.872400 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 04:43:11.872602 3589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.864929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:45591 (local address 127.0.0.1:51390)
0928 04:43:11.865096 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:11.865106 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:11.865190 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:11.865325 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:11.865326 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:11.865639 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:11.865728 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.865731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.866457 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.866458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.866995 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.866997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.867354 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.867355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.867406 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.867983 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:11.867987 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:11.869510 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:11.871369 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:11.871376 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:11.871378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:11.871578 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:11.871809 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:11.871810 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:11.871810 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:11.871872 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:11.872126 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:11.872133 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:11.872213 (+ 80us) client_negotiation.cc:769] Sending connection context
0928 04:43:11.872296 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 04:43:11.872466 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":72,"thread_start_us":99,"threads_started":1}
I20260928 04:43:11.875075 3045 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:51390:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 04:43:11.875303 3045 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:43:11.876737 3454 tablet_service.cc:1467] Processing CreateTablet for tablet f5018df4f67b438bb799f1df3c4eb9b1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a4d40bca0ba9487ba274f372602cc3a8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 04:43:11.876767 3183 tablet_service.cc:1467] Processing CreateTablet for tablet f5018df4f67b438bb799f1df3c4eb9b1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a4d40bca0ba9487ba274f372602cc3a8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 04:43:11.876899 3454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5018df4f67b438bb799f1df3c4eb9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:11.876919 3183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5018df4f67b438bb799f1df3c4eb9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:11.877939 3552 tablet_bootstrap.cc:492] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe: Bootstrap starting.
I20260928 04:43:11.878301 3552 tablet_bootstrap.cc:654] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:11.878626 3315 tablet_service.cc:1467] Processing CreateTablet for tablet f5018df4f67b438bb799f1df3c4eb9b1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a4d40bca0ba9487ba274f372602cc3a8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 04:43:11.878768 3315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5018df4f67b438bb799f1df3c4eb9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:11.879120 3552 tablet_bootstrap.cc:492] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe: No bootstrap required, opened a new log
I20260928 04:43:11.879181 3552 ts_tablet_manager.cc:1397] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:11.879328 3552 raft_consensus.cc:348] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } }
I20260928 04:43:11.879383 3552 raft_consensus.cc:374] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:11.879396 3552 raft_consensus.cc:729] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa371b22cc8498bb8ec4cbbf76fa5fe, State: Initialized, Role: FOLLOWER
I20260928 04:43:11.879446 3552 consensus_queue.cc:260] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } }
I20260928 04:43:11.879534 3552 ts_tablet_manager.cc:1428] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:11.879546 3571 tablet_replica.cc:439] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } } }
I20260928 04:43:11.879643 3571 tablet_replica.cc:442] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:43:11.880234 3556 tablet_bootstrap.cc:492] T f5018df4f67b438bb799f1df3c4eb9b1 P 2e5ffb4c14d347b6abbddd50bc38fb7a: Bootstrap starting.
I20260928 04:43:11.880251 3553 tablet_bootstrap.cc:492] T f5018df4f67b438bb799f1df3c4eb9b1 P 780b2a558b5644c4b7e64975f9293906: Bootstrap starting.
I20260928 04:43:11.880578 3556 tablet_bootstrap.cc:654] T f5018df4f67b438bb799f1df3c4eb9b1 P 2e5ffb4c14d347b6abbddd50bc38fb7a: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:11.880654 3553 tablet_bootstrap.cc:654] T f5018df4f67b438bb799f1df3c4eb9b1 P 780b2a558b5644c4b7e64975f9293906: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:11.881172 3556 tablet_bootstrap.cc:492] T f5018df4f67b438bb799f1df3c4eb9b1 P 2e5ffb4c14d347b6abbddd50bc38fb7a: No bootstrap required, opened a new log
I20260928 04:43:11.881238 3556 ts_tablet_manager.cc:1397] T f5018df4f67b438bb799f1df3c4eb9b1 P 2e5ffb4c14d347b6abbddd50bc38fb7a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 04:43:11.881384 3556 raft_consensus.cc:348] T f5018df4f67b438bb799f1df3c4eb9b1 P 2e5ffb4c14d347b6abbddd50bc38fb7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } }
I20260928 04:43:11.881448 3556 raft_consensus.cc:374] T f5018df4f67b438bb799f1df3c4eb9b1 P 2e5ffb4c14d347b6abbddd50bc38fb7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:11.881471 3556 raft_consensus.cc:729] T f5018df4f67b438bb799f1df3c4eb9b1 P 2e5ffb4c14d347b6abbddd50bc38fb7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e5ffb4c14d347b6abbddd50bc38fb7a, State: Initialized, Role: FOLLOWER
I20260928 04:43:11.881517 3556 consensus_queue.cc:260] T f5018df4f67b438bb799f1df3c4eb9b1 P 2e5ffb4c14d347b6abbddd50bc38fb7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } }
I20260928 04:43:11.881597 3556 ts_tablet_manager.cc:1428] T f5018df4f67b438bb799f1df3c4eb9b1 P 2e5ffb4c14d347b6abbddd50bc38fb7a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:11.881668 3560 tablet_replica.cc:439] T f5018df4f67b438bb799f1df3c4eb9b1 P 2e5ffb4c14d347b6abbddd50bc38fb7a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } } }
I20260928 04:43:11.881775 3560 tablet_replica.cc:442] T f5018df4f67b438bb799f1df3c4eb9b1 P 2e5ffb4c14d347b6abbddd50bc38fb7a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:43:11.882079 3553 tablet_bootstrap.cc:492] T f5018df4f67b438bb799f1df3c4eb9b1 P 780b2a558b5644c4b7e64975f9293906: No bootstrap required, opened a new log
I20260928 04:43:11.882140 3553 ts_tablet_manager.cc:1397] T f5018df4f67b438bb799f1df3c4eb9b1 P 780b2a558b5644c4b7e64975f9293906: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260928 04:43:11.882269 3553 raft_consensus.cc:348] T f5018df4f67b438bb799f1df3c4eb9b1 P 780b2a558b5644c4b7e64975f9293906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } }
I20260928 04:43:11.882340 3553 raft_consensus.cc:374] T f5018df4f67b438bb799f1df3c4eb9b1 P 780b2a558b5644c4b7e64975f9293906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:11.882360 3553 raft_consensus.cc:729] T f5018df4f67b438bb799f1df3c4eb9b1 P 780b2a558b5644c4b7e64975f9293906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 780b2a558b5644c4b7e64975f9293906, State: Initialized, Role: FOLLOWER
I20260928 04:43:11.882404 3553 consensus_queue.cc:260] T f5018df4f67b438bb799f1df3c4eb9b1 P 780b2a558b5644c4b7e64975f9293906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } }
I20260928 04:43:11.882511 3553 ts_tablet_manager.cc:1428] T f5018df4f67b438bb799f1df3c4eb9b1 P 780b2a558b5644c4b7e64975f9293906: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:11.882555 3559 tablet_replica.cc:439] T f5018df4f67b438bb799f1df3c4eb9b1 P 780b2a558b5644c4b7e64975f9293906: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } } }
I20260928 04:43:11.882656 3559 tablet_replica.cc:442] T f5018df4f67b438bb799f1df3c4eb9b1 P 780b2a558b5644c4b7e64975f9293906: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:43:11.951022 3571 raft_consensus.cc:482] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:43:11.951129 3571 raft_consensus.cc:504] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } }
I20260928 04:43:11.951357 3571 leader_election.cc:288] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 780b2a558b5644c4b7e64975f9293906 (127.1.226.193:37671), 2e5ffb4c14d347b6abbddd50bc38fb7a (127.1.226.194:36237)
I20260928 04:43:11.951606 3343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5018df4f67b438bb799f1df3c4eb9b1" candidate_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" is_pre_election: true
I20260928 04:43:11.951604 3203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5018df4f67b438bb799f1df3c4eb9b1" candidate_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "780b2a558b5644c4b7e64975f9293906" is_pre_election: true
I20260928 04:43:11.951738 3203 raft_consensus.cc:2454] T f5018df4f67b438bb799f1df3c4eb9b1 P 780b2a558b5644c4b7e64975f9293906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efa371b22cc8498bb8ec4cbbf76fa5fe in term 0.
I20260928 04:43:11.951745 3343 raft_consensus.cc:2454] T f5018df4f67b438bb799f1df3c4eb9b1 P 2e5ffb4c14d347b6abbddd50bc38fb7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efa371b22cc8498bb8ec4cbbf76fa5fe in term 0.
I20260928 04:43:11.951926 3415 leader_election.cc:302] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e5ffb4c14d347b6abbddd50bc38fb7a, efa371b22cc8498bb8ec4cbbf76fa5fe; no voters:
I20260928 04:43:11.952054 3571 raft_consensus.cc:2781] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:43:11.952098 3571 raft_consensus.cc:482] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:43:11.952112 3571 raft_consensus.cc:3037] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:11.952788 3571 raft_consensus.cc:504] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } }
I20260928 04:43:11.952935 3571 leader_election.cc:288] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe [CANDIDATE]: Term 1 election: Requested vote from peers 780b2a558b5644c4b7e64975f9293906 (127.1.226.193:37671), 2e5ffb4c14d347b6abbddd50bc38fb7a (127.1.226.194:36237)
I20260928 04:43:11.953074 3203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5018df4f67b438bb799f1df3c4eb9b1" candidate_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "780b2a558b5644c4b7e64975f9293906"
I20260928 04:43:11.953101 3343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5018df4f67b438bb799f1df3c4eb9b1" candidate_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a"
I20260928 04:43:11.953163 3343 raft_consensus.cc:3037] T f5018df4f67b438bb799f1df3c4eb9b1 P 2e5ffb4c14d347b6abbddd50bc38fb7a [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:11.953162 3203 raft_consensus.cc:3037] T f5018df4f67b438bb799f1df3c4eb9b1 P 780b2a558b5644c4b7e64975f9293906 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:11.953704 3203 raft_consensus.cc:2454] T f5018df4f67b438bb799f1df3c4eb9b1 P 780b2a558b5644c4b7e64975f9293906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efa371b22cc8498bb8ec4cbbf76fa5fe in term 1.
I20260928 04:43:11.953848 3343 raft_consensus.cc:2454] T f5018df4f67b438bb799f1df3c4eb9b1 P 2e5ffb4c14d347b6abbddd50bc38fb7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efa371b22cc8498bb8ec4cbbf76fa5fe in term 1.
I20260928 04:43:11.953912 3417 leader_election.cc:302] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 780b2a558b5644c4b7e64975f9293906, efa371b22cc8498bb8ec4cbbf76fa5fe; no voters:
I20260928 04:43:11.954035 3571 raft_consensus.cc:2781] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:11.954101 3571 raft_consensus.cc:686] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe [term 1 LEADER]: Becoming Leader. State: Replica: efa371b22cc8498bb8ec4cbbf76fa5fe, State: Running, Role: LEADER
I20260928 04:43:11.954159 3571 consensus_queue.cc:237] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } }
I20260928 04:43:11.954471 3562 tablet_replica.cc:439] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe: TxnStatusTablet state changed. Reason: New leader efa371b22cc8498bb8ec4cbbf76fa5fe. Latest consensus state: current_term: 1 leader_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } } }
I20260928 04:43:11.954558 3562 tablet_replica.cc:442] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe: This TxnStatusTablet replica's current role is: LEADER
I20260928 04:43:11.954734 3594 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 04:43:11.954854 3594 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 04:43:11.954949 3045 catalog_manager.cc:5495] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe reported cstate change: term changed from 0 to 1, leader changed from <none> to efa371b22cc8498bb8ec4cbbf76fa5fe (127.1.226.195). New cstate: current_term: 1 leader_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } health_report { overall_health: HEALTHY } } }
I20260928 04:43:11.978971 3549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.976314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40184 (local address 127.1.226.195:39555)
0928 04:43:11.976385 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:11.976387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:11.976465 (+ 78us) server_negotiation.cc:407] Connection header received
0928 04:43:11.976495 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:11.976496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:11.976510 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:11.976577 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:43:11.976769 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.977244 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.978348 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.978521 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.978566 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.978751 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:43:11.978776 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:43:11.978818 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:11.978895 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 04:43:11.978912 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 04:43:11.979022 3589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.976230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:39555 (local address 127.0.0.1:40184)
0928 04:43:11.976317 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:11.976329 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:11.976373 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:11.976601 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:11.976602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:11.976604 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:43:11.976676 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.976679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.977334 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.977336 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.978259 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.978262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.978582 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.978584 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.978649 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.978669 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:43:11.978834 (+ 165us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:43:11.978841 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 04:43:11.978894 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 04:43:11.978939 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 04:43:11.980001 3203 raft_consensus.cc:1261] T f5018df4f67b438bb799f1df3c4eb9b1 P 780b2a558b5644c4b7e64975f9293906 [term 1 FOLLOWER]: Refusing update from remote peer efa371b22cc8498bb8ec4cbbf76fa5fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:11.980038 3343 raft_consensus.cc:1261] T f5018df4f67b438bb799f1df3c4eb9b1 P 2e5ffb4c14d347b6abbddd50bc38fb7a [term 1 FOLLOWER]: Refusing update from remote peer efa371b22cc8498bb8ec4cbbf76fa5fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:11.980280 3562 consensus_queue.cc:1035] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:11.980489 3562 consensus_queue.cc:1035] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:11.980851 3559 tablet_replica.cc:439] T f5018df4f67b438bb799f1df3c4eb9b1 P 780b2a558b5644c4b7e64975f9293906: TxnStatusTablet state changed. Reason: New leader efa371b22cc8498bb8ec4cbbf76fa5fe. Latest consensus state: current_term: 1 leader_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } } }
I20260928 04:43:11.980952 3559 tablet_replica.cc:442] T f5018df4f67b438bb799f1df3c4eb9b1 P 780b2a558b5644c4b7e64975f9293906: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:43:11.981426 3560 tablet_replica.cc:439] T f5018df4f67b438bb799f1df3c4eb9b1 P 2e5ffb4c14d347b6abbddd50bc38fb7a: TxnStatusTablet state changed. Reason: New leader efa371b22cc8498bb8ec4cbbf76fa5fe. Latest consensus state: current_term: 1 leader_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } } }
I20260928 04:43:11.981525 3560 tablet_replica.cc:442] T f5018df4f67b438bb799f1df3c4eb9b1 P 2e5ffb4c14d347b6abbddd50bc38fb7a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:43:11.981637 3571 tablet_replica.cc:439] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } } }
I20260928 04:43:11.981750 3571 tablet_replica.cc:442] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe: This TxnStatusTablet replica's current role is: LEADER
I20260928 04:43:11.981809 3571 tablet_replica.cc:439] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } } }
I20260928 04:43:11.981855 3571 tablet_replica.cc:442] T f5018df4f67b438bb799f1df3c4eb9b1 P efa371b22cc8498bb8ec4cbbf76fa5fe: This TxnStatusTablet replica's current role is: LEADER
I20260928 04:43:11.981853 3559 tablet_replica.cc:439] T f5018df4f67b438bb799f1df3c4eb9b1 P 780b2a558b5644c4b7e64975f9293906: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } } }
I20260928 04:43:11.981922 3559 tablet_replica.cc:442] T f5018df4f67b438bb799f1df3c4eb9b1 P 780b2a558b5644c4b7e64975f9293906: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:43:11.982551 3560 tablet_replica.cc:439] T f5018df4f67b438bb799f1df3c4eb9b1 P 2e5ffb4c14d347b6abbddd50bc38fb7a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 } } peers { permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 } } peers { permanent_uuid: "efa371b22cc8498bb8ec4cbbf76fa5fe" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39555 } } }
I20260928 04:43:11.982627 3560 tablet_replica.cc:442] T f5018df4f67b438bb799f1df3c4eb9b1 P 2e5ffb4c14d347b6abbddd50bc38fb7a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:43:11.988497 3535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.986132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:39555 (local address 127.1.226.195:43795)
0928 04:43:11.986237 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:11.986248 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:11.986317 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:11.986432 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:11.986433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:11.986434 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:43:11.986481 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.986482 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.987064 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.987065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.987684 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:11.987686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.988043 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.988044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:11.988097 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.988110 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:43:11.988314 (+ 204us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:43:11.988316 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:43:11.988362 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 04:43:11.988407 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":8}
I20260928 04:43:11.988497 3549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:11.986181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:43795 (local address 127.1.226.195:39555)
0928 04:43:11.986220 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:11.986222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:11.986286 (+ 64us) server_negotiation.cc:407] Connection header received
0928 04:43:11.986364 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:11.986365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:11.986377 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:11.986429 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:43:11.986536 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.986990 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.987788 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:11.987970 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:11.988041 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:11.988221 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:43:11.988256 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:43:11.988314 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:11.988362 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 04:43:11.988389 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":15}
I20260928 04:43:11.990593 3343 raft_consensus.cc:1261] T f9e04284c5de43a293042d6f221e05dc P 2e5ffb4c14d347b6abbddd50bc38fb7a [term 1 FOLLOWER]: Refusing update from remote peer efa371b22cc8498bb8ec4cbbf76fa5fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:11.990609 3203 raft_consensus.cc:1261] T f9e04284c5de43a293042d6f221e05dc P 780b2a558b5644c4b7e64975f9293906 [term 1 FOLLOWER]: Refusing update from remote peer efa371b22cc8498bb8ec4cbbf76fa5fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:11.990808 3562 consensus_queue.cc:1035] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e5ffb4c14d347b6abbddd50bc38fb7a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:11.990912 3571 consensus_queue.cc:1035] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "780b2a558b5644c4b7e64975f9293906" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:12.001515 3045 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:43:12.001618 3045 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:43:12.002333 3045 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2bec407195a84052afacd9c2624e3bc0: Sending DeleteTablet for 3 replicas of tablet f9e04284c5de43a293042d6f221e05dc
I20260928 04:43:12.002626 3315 tablet_service.cc:1514] Processing DeleteTablet for tablet f9e04284c5de43a293042d6f221e05dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:11 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:54168
I20260928 04:43:12.002746 3183 tablet_service.cc:1514] Processing DeleteTablet for tablet f9e04284c5de43a293042d6f221e05dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:11 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:37294
I20260928 04:43:12.002755 3575 tablet_replica.cc:330] T f9e04284c5de43a293042d6f221e05dc P 2e5ffb4c14d347b6abbddd50bc38fb7a: stopping tablet replica
I20260928 04:43:12.002818 3575 raft_consensus.cc:2229] T f9e04284c5de43a293042d6f221e05dc P 2e5ffb4c14d347b6abbddd50bc38fb7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:43:12.002856 3575 raft_consensus.cc:2258] T f9e04284c5de43a293042d6f221e05dc P 2e5ffb4c14d347b6abbddd50bc38fb7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:12.002930 3577 tablet_replica.cc:330] T f9e04284c5de43a293042d6f221e05dc P 780b2a558b5644c4b7e64975f9293906: stopping tablet replica
I20260928 04:43:12.002987 3577 raft_consensus.cc:2229] T f9e04284c5de43a293042d6f221e05dc P 780b2a558b5644c4b7e64975f9293906 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:43:12.003022 3577 raft_consensus.cc:2258] T f9e04284c5de43a293042d6f221e05dc P 780b2a558b5644c4b7e64975f9293906 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:12.003171 3463 tablet_service.cc:1514] Processing DeleteTablet for tablet f9e04284c5de43a293042d6f221e05dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:11 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:40160
I20260928 04:43:12.003289 3576 tablet_replica.cc:330] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe: stopping tablet replica
I20260928 04:43:12.003360 3576 raft_consensus.cc:2229] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:43:12.003419 3576 raft_consensus.cc:2258] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:12.003566 3577 ts_tablet_manager.cc:1905] T f9e04284c5de43a293042d6f221e05dc P 780b2a558b5644c4b7e64975f9293906: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:12.003731 3576 ts_tablet_manager.cc:1905] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:12.003810 3575 ts_tablet_manager.cc:1905] T f9e04284c5de43a293042d6f221e05dc P 2e5ffb4c14d347b6abbddd50bc38fb7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:12.005610 3576 ts_tablet_manager.cc:1918] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:43:12.005636 3577 ts_tablet_manager.cc:1918] T f9e04284c5de43a293042d6f221e05dc P 780b2a558b5644c4b7e64975f9293906: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:43:12.005682 3576 log.cc:1196] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-2/wal/wals/f9e04284c5de43a293042d6f221e05dc
I20260928 04:43:12.005700 3577 log.cc:1196] T f9e04284c5de43a293042d6f221e05dc P 780b2a558b5644c4b7e64975f9293906: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-0/wal/wals/f9e04284c5de43a293042d6f221e05dc
I20260928 04:43:12.005923 3576 ts_tablet_manager.cc:1939] T f9e04284c5de43a293042d6f221e05dc P efa371b22cc8498bb8ec4cbbf76fa5fe: Deleting consensus metadata
I20260928 04:43:12.005923 3577 ts_tablet_manager.cc:1939] T f9e04284c5de43a293042d6f221e05dc P 780b2a558b5644c4b7e64975f9293906: Deleting consensus metadata
I20260928 04:43:12.006137 3575 ts_tablet_manager.cc:1918] T f9e04284c5de43a293042d6f221e05dc P 2e5ffb4c14d347b6abbddd50bc38fb7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:43:12.006206 3575 log.cc:1196] T f9e04284c5de43a293042d6f221e05dc P 2e5ffb4c14d347b6abbddd50bc38fb7a: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570585693787-1931-0/minicluster-data/ts-1/wal/wals/f9e04284c5de43a293042d6f221e05dc
I20260928 04:43:12.006414 3575 ts_tablet_manager.cc:1939] T f9e04284c5de43a293042d6f221e05dc P 2e5ffb4c14d347b6abbddd50bc38fb7a: Deleting consensus metadata
I20260928 04:43:12.006418 3032 catalog_manager.cc:4841] TS efa371b22cc8498bb8ec4cbbf76fa5fe (127.1.226.195:39555): tablet f9e04284c5de43a293042d6f221e05dc (table test-table [id=c16b589b55b14ebaa7830b75a8a8aaef]) successfully deleted
I20260928 04:43:12.006743 3031 catalog_manager.cc:4841] TS 2e5ffb4c14d347b6abbddd50bc38fb7a (127.1.226.194:36237): tablet f9e04284c5de43a293042d6f221e05dc (table test-table [id=c16b589b55b14ebaa7830b75a8a8aaef]) successfully deleted
I20260928 04:43:12.009402 3033 catalog_manager.cc:4841] TS 780b2a558b5644c4b7e64975f9293906 (127.1.226.193:37671): tablet f9e04284c5de43a293042d6f221e05dc (table test-table [id=c16b589b55b14ebaa7830b75a8a8aaef]) successfully deleted
Sep 28 04:43:12 dist-test-slave-j5c3 krb5kdc[3001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570591, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.226.193@KRBTEST.COM
I20260928 04:43:12.015501 3551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:12.005622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37304 (local address 127.1.226.193:37671)
0928 04:43:12.005671 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:12.005673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:12.005859 (+ 186us) server_negotiation.cc:407] Connection header received
0928 04:43:12.005934 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:12.005935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:12.005946 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:12.005992 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:12.006963 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:12.007452 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:12.008190 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:12.008404 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:12.008491 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:12.011759 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:12.011771 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:12.011772 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:12.011780 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:12.014061 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:12.014477 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:12.014481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:12.014481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:12.014547 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:12.014945 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:12.014947 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:12.014947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:12.015115 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:12.015240 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:12.015361 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 04:43:12.015431 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 04:43:12.016290 3243 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37304
I20260928 04:43:12.023823 3112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:12.017359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51406 (local address 127.1.226.254:45591)
0928 04:43:12.017404 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:12.017406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:12.017443 (+ 37us) server_negotiation.cc:407] Connection header received
0928 04:43:12.017521 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:12.017522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:12.017531 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:12.017577 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:12.018219 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:12.018740 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:12.019747 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:12.019943 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:12.019996 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:12.021366 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:12.021368 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:12.021368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:12.021375 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:12.022750 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:12.023139 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:12.023139 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:12.023140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:12.023189 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:12.023388 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:12.023388 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:12.023389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:12.023490 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:12.023567 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:12.023683 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 04:43:12.023745 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 04:43:12.024298 3045 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51406
I20260928 04:43:12.025410 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 3118
I20260928 04:43:12.030875 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 3258
I20260928 04:43:12.035760 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 3398
I20260928 04:43:12.041764 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 3014
2026-09-28T04:43:12Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2380 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:43:12 dist-test-slave-j5c3 krb5kdc[3609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:43:12 dist-test-slave-j5c3 krb5kdc[3609](info): set up 2 sockets
Sep 28 04:43:12 dist-test-slave-j5c3 krb5kdc[3609](info): commencing operation
krb5kdc: starting...
W20260928 04:43:13.076016 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:43:13 dist-test-slave-j5c3 krb5kdc[3609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:43:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:43:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:43:13.199128 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:41127
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:41127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:13.283077 3622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:13.283332 3622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:13.283373 3622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:13.284600 3622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:43:13.284679 3622 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 04:43:13.284700 3622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:13.284722 3622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:13.284773 3622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:43:13.284794 3622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:43:13.286412 3622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:41127
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:41127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:13.286721 3622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:13.286953 3622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:13.292061 3627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:13.292053 3630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:13.292202 3622 server_base.cc:1034] running on GCE node
W20260928 04:43:13.292361 3628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:13.292629 3622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:13.292906 3622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:13.294072 3622 hybrid_clock.cc:648] HybridClock initialized: now 1790570593294060 us; error 32 us; skew 500 ppm
Sep 28 04:43:13 dist-test-slave-j5c3 krb5kdc[3609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570593, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:13.296509 3622 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:43:13.296972 3622 webserver.cc:460] Webserver started at http://127.1.226.254:36303/ using document root <none> and password file <none>
I20260928 04:43:13.297207 3622 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:13.297272 3622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:13.297413 3622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:13.298738 3622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "6955b3f0c50e4c0cb1864be5c8d8eb41"
format_stamp: "Formatted at 2026-09-28 04:43:13 on dist-test-slave-j5c3"
server_key: "6f14c8e9ab23a80efd17f4ecd0f6dba4"
server_key_iv: "f450f63501e42b66b5ff6b733461650c"
server_key_version: "encryptionkey@0"
I20260928 04:43:13.299280 3622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "6955b3f0c50e4c0cb1864be5c8d8eb41"
format_stamp: "Formatted at 2026-09-28 04:43:13 on dist-test-slave-j5c3"
server_key: "6f14c8e9ab23a80efd17f4ecd0f6dba4"
server_key_iv: "f450f63501e42b66b5ff6b733461650c"
server_key_version: "encryptionkey@0"
I20260928 04:43:13.300776 3622 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260928 04:43:13.301519 3637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:13.301704 3622 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:43:13.301805 3622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "6955b3f0c50e4c0cb1864be5c8d8eb41"
format_stamp: "Formatted at 2026-09-28 04:43:13 on dist-test-slave-j5c3"
server_key: "6f14c8e9ab23a80efd17f4ecd0f6dba4"
server_key_iv: "f450f63501e42b66b5ff6b733461650c"
server_key_version: "encryptionkey@0"
I20260928 04:43:13.301890 3622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:13.321398 3622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:13.323422 3622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:13.323585 3622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:13.328377 3622 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:41127
I20260928 04:43:13.328414 3699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:41127 every 8 connection(s)
I20260928 04:43:13.328750 3622 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:43:13.329413 3700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:13.331723 3700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6955b3f0c50e4c0cb1864be5c8d8eb41: Bootstrap starting.
I20260928 04:43:13.332327 3700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6955b3f0c50e4c0cb1864be5c8d8eb41: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:13.332623 3700 log.cc:824] T 00000000000000000000000000000000 P 6955b3f0c50e4c0cb1864be5c8d8eb41: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:13.333300 3700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6955b3f0c50e4c0cb1864be5c8d8eb41: No bootstrap required, opened a new log
I20260928 04:43:13.334241 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 3622
I20260928 04:43:13.334371 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:43:13.334533 1931 external_mini_cluster.cc:1428] Setting key 453ee2c381098224d73ddec6fadcf18e
I20260928 04:43:13.335505 3700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6955b3f0c50e4c0cb1864be5c8d8eb41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6955b3f0c50e4c0cb1864be5c8d8eb41" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 41127 } }
I20260928 04:43:13.335729 3700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6955b3f0c50e4c0cb1864be5c8d8eb41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:13.335760 3700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6955b3f0c50e4c0cb1864be5c8d8eb41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6955b3f0c50e4c0cb1864be5c8d8eb41, State: Initialized, Role: FOLLOWER
I20260928 04:43:13.335927 3700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6955b3f0c50e4c0cb1864be5c8d8eb41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6955b3f0c50e4c0cb1864be5c8d8eb41" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 41127 } }
I20260928 04:43:13.336004 3700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6955b3f0c50e4c0cb1864be5c8d8eb41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:43:13.336052 3700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6955b3f0c50e4c0cb1864be5c8d8eb41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:43:13.336107 3700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6955b3f0c50e4c0cb1864be5c8d8eb41 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:13.336750 3700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6955b3f0c50e4c0cb1864be5c8d8eb41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6955b3f0c50e4c0cb1864be5c8d8eb41" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 41127 } }
I20260928 04:43:13.336894 3700 leader_election.cc:302] T 00000000000000000000000000000000 P 6955b3f0c50e4c0cb1864be5c8d8eb41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6955b3f0c50e4c0cb1864be5c8d8eb41; no voters:
I20260928 04:43:13.337081 3700 leader_election.cc:288] T 00000000000000000000000000000000 P 6955b3f0c50e4c0cb1864be5c8d8eb41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:43:13.337127 3705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6955b3f0c50e4c0cb1864be5c8d8eb41 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:13.337255 3705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6955b3f0c50e4c0cb1864be5c8d8eb41 [term 1 LEADER]: Becoming Leader. State: Replica: 6955b3f0c50e4c0cb1864be5c8d8eb41, State: Running, Role: LEADER
I20260928 04:43:13.337350 3700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6955b3f0c50e4c0cb1864be5c8d8eb41 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:43:13.337342 3705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6955b3f0c50e4c0cb1864be5c8d8eb41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6955b3f0c50e4c0cb1864be5c8d8eb41" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 41127 } }
I20260928 04:43:13.338562 3706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6955b3f0c50e4c0cb1864be5c8d8eb41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6955b3f0c50e4c0cb1864be5c8d8eb41. Latest consensus state: current_term: 1 leader_uuid: "6955b3f0c50e4c0cb1864be5c8d8eb41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6955b3f0c50e4c0cb1864be5c8d8eb41" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 41127 } } }
I20260928 04:43:13.338666 3706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6955b3f0c50e4c0cb1864be5c8d8eb41 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:13.338925 3719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:43:13.339383 3719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:43:13.338739 3707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6955b3f0c50e4c0cb1864be5c8d8eb41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6955b3f0c50e4c0cb1864be5c8d8eb41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6955b3f0c50e4c0cb1864be5c8d8eb41" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 41127 } } }
I20260928 04:43:13.340785 3707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6955b3f0c50e4c0cb1864be5c8d8eb41 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:13.341182 3719 catalog_manager.cc:1347] Generated new cluster ID: bf310e16bdaa4574a56841e1b275cfb6
I20260928 04:43:13.341238 3719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:43:13 dist-test-slave-j5c3 krb5kdc[3609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:13.346172 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:13.335337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48068 (local address 127.1.226.254:41127)
0928 04:43:13.335517 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:13.335525 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:13.335550 (+ 25us) server_negotiation.cc:407] Connection header received
0928 04:43:13.335641 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:13.335646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:13.335688 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:13.335883 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:13.336684 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.337920 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.338837 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.339385 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.339494 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:13.342199 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:13.342202 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:13.342203 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:13.342216 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:13.344300 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:13.344684 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:13.344687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:13.344687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:13.344828 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:13.345035 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:13.345036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:13.345036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:13.345364 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:13.345534 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:13.345626 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 04:43:13.345777 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
I20260928 04:43:13.351354 3719 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:43:13.351856 3719 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:43:13.358940 3719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6955b3f0c50e4c0cb1864be5c8d8eb41: Generated new TSK 0
I20260928 04:43:13.359133 3719 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.226.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
I20260928 04:43:13.386523 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:0
--local_ip_for_outbound_sockets=127.1.226.193
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:41127
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:13.470398 3726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:13.470645 3726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:13.470710 3726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:13.472005 3726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:13.472051 3726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:13.472075 3726 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 04:43:13.472090 3726 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 04:43:13.472108 3726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:43:13.473724 3726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.226.254:41127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:13.473922 3726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:13.474112 3726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:13.479596 3731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:13.479576 3732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:13.479573 3734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:13.479612 3726 server_base.cc:1034] running on GCE node
I20260928 04:43:13.479992 3726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:13.480302 3726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:13.481457 3726 hybrid_clock.cc:648] HybridClock initialized: now 1790570593481445 us; error 40 us; skew 500 ppm
Sep 28 04:43:13 dist-test-slave-j5c3 krb5kdc[3609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570593, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:13.483915 3726 init.cc:377] Logged in from keytab as kudu/127.1.226.193@KRBTEST.COM (short username kudu)
I20260928 04:43:13.484396 3726 webserver.cc:460] Webserver started at http://127.1.226.193:33275/ using document root <none> and password file <none>
I20260928 04:43:13.484613 3726 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:13.484674 3726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:13.484803 3726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:13.485846 3726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-0/data/instance:
uuid: "5c39d49bf2a04951a2094251619c837d"
format_stamp: "Formatted at 2026-09-28 04:43:13 on dist-test-slave-j5c3"
server_key: "54a69343cac13b22d9e84491cbecd053"
server_key_iv: "0fe469107018c9f4fba524e51f5802d4"
server_key_version: "encryptionkey@0"
I20260928 04:43:13.486199 3726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance:
uuid: "5c39d49bf2a04951a2094251619c837d"
format_stamp: "Formatted at 2026-09-28 04:43:13 on dist-test-slave-j5c3"
server_key: "54a69343cac13b22d9e84491cbecd053"
server_key_iv: "0fe469107018c9f4fba524e51f5802d4"
server_key_version: "encryptionkey@0"
I20260928 04:43:13.487510 3726 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:43:13.488337 3741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:13.488507 3726 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:43:13.488600 3726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "5c39d49bf2a04951a2094251619c837d"
format_stamp: "Formatted at 2026-09-28 04:43:13 on dist-test-slave-j5c3"
server_key: "54a69343cac13b22d9e84491cbecd053"
server_key_iv: "0fe469107018c9f4fba524e51f5802d4"
server_key_version: "encryptionkey@0"
I20260928 04:43:13.488675 3726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:13.498939 3726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:13.501086 3726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:13.501255 3726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:13.501814 3726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:13.501871 3726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:13.501924 3726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:13.501946 3726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:13.508989 3726 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:41701
I20260928 04:43:13.509066 3856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:41701 every 8 connection(s)
Sep 28 04:43:13 dist-test-slave-j5c3 krb5kdc[3609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570593, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:13.509372 3726 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
I20260928 04:43:13.511510 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 3726
I20260928 04:43:13.511627 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance
I20260928 04:43:13.511916 1931 external_mini_cluster.cc:1428] Setting key 7e8cb969e0eb1108f3c26ebbe1c6fa79
I20260928 04:43:13.512842 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:13.502405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:35791 (local address 127.1.226.254:41127)
0928 04:43:13.502462 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:13.502464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:13.502737 (+ 273us) server_negotiation.cc:407] Connection header received
0928 04:43:13.503408 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:13.503411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:13.503429 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:13.503500 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:13.504478 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.505336 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.506585 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.506858 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.506938 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:13.509890 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:13.509891 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:13.509891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:13.509899 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:13.511129 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:13.511685 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:13.511687 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:13.511687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:13.511759 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:13.512064 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:13.512065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:13.512066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:13.512271 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:13.512344 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:13.512640 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 04:43:13.512741 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 04:43:13.513056 3750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:13.502395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:41127 (local address 127.1.226.193:35791)
0928 04:43:13.502568 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:13.502605 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:13.503263 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:13.503614 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:13.503620 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:13.504025 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:13.504340 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:13.504346 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.505453 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.505454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:13.506452 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:13.506456 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.506964 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.506965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:13.507154 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:13.507738 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:13.507748 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:13.509789 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:13.511220 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:13.511227 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:13.511228 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:13.511575 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:13.511832 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:13.511833 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:13.511833 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:13.511941 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:13.512347 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:13.512354 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:13.512499 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 04:43:13.512617 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 04:43:13.512801 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
I20260928 04:43:13.513551 3857 heartbeater.cc:347] Connected to a master server at 127.1.226.254:41127
I20260928 04:43:13.513646 3857 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:13.513841 3857 heartbeater.cc:511] Master 127.1.226.254:41127 requested a full tablet report, sending...
I20260928 04:43:13.514380 3653 ts_manager.cc:194] Registered new tserver with Master: 5c39d49bf2a04951a2094251619c837d (127.1.226.193:41701)
I20260928 04:43:13.515172 3653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.193@KRBTEST.COM'} at 127.1.226.193:35791
I20260928 04:43:13.520447 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:13.514892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:55963 (local address 127.1.226.254:41127)
0928 04:43:13.514934 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:13.514936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:13.514948 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:43:13.514981 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:13.514982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:13.514992 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:13.515064 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:13.515552 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.516023 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.516759 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.516967 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.517050 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:13.517961 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:13.517962 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:13.517963 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:13.517969 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:13.519311 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:13.519668 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:13.519669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:13.519670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:13.519723 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:13.519917 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:13.519918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:13.519918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:13.520027 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:13.520098 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:13.520270 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 04:43:13.520365 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:43:13.520455 3864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:13.514716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:41127 (local address 127.1.226.193:55963)
0928 04:43:13.514877 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:13.514890 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:13.514940 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:13.515067 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:13.515068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:13.515321 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:13.515469 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:13.515471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.516110 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.516110 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:13.516681 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:13.516683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.517051 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.517051 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:13.517128 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:13.517572 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:13.517577 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:13.517861 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:13.519404 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:13.519405 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:13.519405 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:13.519575 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:13.519789 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:13.519790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:13.519790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:13.519853 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:13.520089 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:13.520090 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:13.520136 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 04:43:13.520235 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 04:43:13.520342 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for kudu/127.1.226.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
I20260928 04:43:13.539026 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:0
--local_ip_for_outbound_sockets=127.1.226.194
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:41127
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:13.629693 3866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:13.629921 3866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:13.629983 3866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:13.631343 3866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:13.631390 3866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:13.631412 3866 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 04:43:13.631423 3866 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 04:43:13.631440 3866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:43:13.632898 3866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.226.254:41127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:13.633095 3866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:13.633284 3866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:13.638518 3874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:13.638521 3872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:13.638566 3871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:13.638926 3866 server_base.cc:1034] running on GCE node
I20260928 04:43:13.639083 3866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:13.639292 3866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:13.640424 3866 hybrid_clock.cc:648] HybridClock initialized: now 1790570593640414 us; error 31 us; skew 500 ppm
Sep 28 04:43:13 dist-test-slave-j5c3 krb5kdc[3609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570593, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:13.642940 3866 init.cc:377] Logged in from keytab as kudu/127.1.226.194@KRBTEST.COM (short username kudu)
I20260928 04:43:13.643388 3866 webserver.cc:460] Webserver started at http://127.1.226.194:40641/ using document root <none> and password file <none>
I20260928 04:43:13.643626 3866 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:13.643693 3866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:13.643838 3866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:13.644985 3866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-1/data/instance:
uuid: "f881bfbdccd44104bddfce14b5d0ba2a"
format_stamp: "Formatted at 2026-09-28 04:43:13 on dist-test-slave-j5c3"
server_key: "4c78d2245260ebd64a884c51c86c76cd"
server_key_iv: "d1fb05cd63956e25d614d134d339c6d8"
server_key_version: "encryptionkey@0"
I20260928 04:43:13.645351 3866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance:
uuid: "f881bfbdccd44104bddfce14b5d0ba2a"
format_stamp: "Formatted at 2026-09-28 04:43:13 on dist-test-slave-j5c3"
server_key: "4c78d2245260ebd64a884c51c86c76cd"
server_key_iv: "d1fb05cd63956e25d614d134d339c6d8"
server_key_version: "encryptionkey@0"
I20260928 04:43:13.646756 3866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:43:13.647516 3881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:13.647713 3866 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:43:13.647884 3866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "f881bfbdccd44104bddfce14b5d0ba2a"
format_stamp: "Formatted at 2026-09-28 04:43:13 on dist-test-slave-j5c3"
server_key: "4c78d2245260ebd64a884c51c86c76cd"
server_key_iv: "d1fb05cd63956e25d614d134d339c6d8"
server_key_version: "encryptionkey@0"
I20260928 04:43:13.647979 3866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:13.665009 3866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:13.667260 3866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:13.667414 3866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:13.667973 3866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:13.668028 3866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:13.668071 3866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:13.668095 3866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 04:43:13 dist-test-slave-j5c3 krb5kdc[3609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570593, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:13.675242 3866 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:43337
I20260928 04:43:13.675475 3996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:43337 every 8 connection(s)
I20260928 04:43:13.675678 3866 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
I20260928 04:43:13.677898 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:13.668353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:60281 (local address 127.1.226.254:41127)
0928 04:43:13.668419 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:13.668422 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:13.668710 (+ 288us) server_negotiation.cc:407] Connection header received
0928 04:43:13.669256 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:13.669258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:13.669271 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:13.669334 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:13.669974 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.670502 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.671534 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.671738 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.671814 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:13.674831 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:13.674833 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:13.674834 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:13.674842 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:13.676468 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:13.676972 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:13.676974 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:13.676974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:13.677042 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:13.677315 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:13.677317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:13.677317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:13.677442 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:13.677510 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:13.677765 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 04:43:13.677836 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 04:43:13.678088 3890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:13.668382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:41127 (local address 127.1.226.194:60281)
0928 04:43:13.668575 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:13.668617 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:13.669142 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:13.669443 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:13.669448 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:13.669693 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:13.669885 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:13.669889 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.670607 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.670608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:13.671435 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:13.671436 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.671823 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.671824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:13.671971 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:13.672515 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:13.672523 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:13.674730 (+ 2207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:13.676554 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:13.676561 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:13.676562 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:13.676884 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:13.677109 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:13.677109 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:13.677110 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:13.677239 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:13.677508 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:13.677513 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:13.677629 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 04:43:13.677736 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 04:43:13.677861 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":120,"threads_started":1}
I20260928 04:43:13.678609 3997 heartbeater.cc:347] Connected to a master server at 127.1.226.254:41127
I20260928 04:43:13.678694 3997 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:13.678896 3997 heartbeater.cc:511] Master 127.1.226.254:41127 requested a full tablet report, sending...
I20260928 04:43:13.679279 3653 ts_manager.cc:194] Registered new tserver with Master: f881bfbdccd44104bddfce14b5d0ba2a (127.1.226.194:43337)
I20260928 04:43:13.679697 3653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.194@KRBTEST.COM'} at 127.1.226.194:60281
I20260928 04:43:13.684732 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 3866
I20260928 04:43:13.684831 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance
I20260928 04:43:13.684924 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:13.679803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:58165 (local address 127.1.226.254:41127)
0928 04:43:13.679838 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:13.679840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:13.679868 (+ 28us) server_negotiation.cc:407] Connection header received
0928 04:43:13.679938 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:13.679939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:13.679949 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:13.680005 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:13.680398 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.680868 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.681559 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.681761 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.681846 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:13.682661 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:13.682662 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:13.682662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:13.682669 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:13.683782 (+ 1113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:13.684127 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:13.684127 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:13.684128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:13.684194 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:13.684391 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:13.684391 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:13.684392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:13.684530 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:13.684599 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:13.684762 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 04:43:13.684837 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 04:43:13.684938 4003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:13.679605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:41127 (local address 127.1.226.194:58165)
0928 04:43:13.679805 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:13.679818 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:13.679881 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:13.680011 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:13.680011 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:13.680243 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:13.680325 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:13.680326 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.680954 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.680955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:13.681481 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:13.681483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.681847 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.681848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:13.681921 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:13.682287 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:13.682305 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:13.682578 (+ 273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:13.683896 (+ 1318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:13.683897 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:13.683898 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:13.684059 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:13.684261 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:13.684262 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:13.684262 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:13.684327 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:13.684602 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:13.684603 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:13.684659 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 04:43:13.684737 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 04:43:13.684836 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":142,"threads_started":1}
I20260928 04:43:13.685035 1931 external_mini_cluster.cc:1428] Setting key 6652f80e784ac1fc60a2667be2465ce7
WARNING: no policy specified for kudu/127.1.226.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
I20260928 04:43:13.711099 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:0
--local_ip_for_outbound_sockets=127.1.226.195
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:41127
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:13.793251 4006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:13.793470 4006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:13.793495 4006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:13.794647 4006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:13.794690 4006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:13.794711 4006 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 04:43:13.794761 4006 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 04:43:13.794800 4006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:43:13.796319 4006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.226.254:41127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:13.796562 4006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:13.796800 4006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:13.801713 4012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:13.801723 4011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:13.801810 4014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:13.802037 4006 server_base.cc:1034] running on GCE node
I20260928 04:43:13.802227 4006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:13.802419 4006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:13.803576 4006 hybrid_clock.cc:648] HybridClock initialized: now 1790570593803565 us; error 36 us; skew 500 ppm
Sep 28 04:43:13 dist-test-slave-j5c3 krb5kdc[3609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570593, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:13.806201 4006 init.cc:377] Logged in from keytab as kudu/127.1.226.195@KRBTEST.COM (short username kudu)
I20260928 04:43:13.806684 4006 webserver.cc:460] Webserver started at http://127.1.226.195:43681/ using document root <none> and password file <none>
I20260928 04:43:13.806903 4006 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:13.806967 4006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:13.807116 4006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:13.808214 4006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-2/data/instance:
uuid: "0ac7e274cd244f9c8ab79ab71210daf4"
format_stamp: "Formatted at 2026-09-28 04:43:13 on dist-test-slave-j5c3"
server_key: "e1434bdbee82d27e02d4d899a6edc89d"
server_key_iv: "898a70a7bfd8a55bb5c574fd62488b60"
server_key_version: "encryptionkey@0"
I20260928 04:43:13.808593 4006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance:
uuid: "0ac7e274cd244f9c8ab79ab71210daf4"
format_stamp: "Formatted at 2026-09-28 04:43:13 on dist-test-slave-j5c3"
server_key: "e1434bdbee82d27e02d4d899a6edc89d"
server_key_iv: "898a70a7bfd8a55bb5c574fd62488b60"
server_key_version: "encryptionkey@0"
I20260928 04:43:13.809911 4006 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:13.810600 4021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:13.810762 4006 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:43:13.810855 4006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "0ac7e274cd244f9c8ab79ab71210daf4"
format_stamp: "Formatted at 2026-09-28 04:43:13 on dist-test-slave-j5c3"
server_key: "e1434bdbee82d27e02d4d899a6edc89d"
server_key_iv: "898a70a7bfd8a55bb5c574fd62488b60"
server_key_version: "encryptionkey@0"
I20260928 04:43:13.810930 4006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:13.840552 4006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:13.843022 4006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:13.843182 4006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:13.843770 4006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:13.843815 4006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:13.843853 4006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:13.843878 4006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:13.850729 4006 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:43749
I20260928 04:43:13.851119 4006 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
Sep 28 04:43:13 dist-test-slave-j5c3 krb5kdc[3609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570593, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:13.851478 4136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:43749 every 8 connection(s)
I20260928 04:43:13.854578 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:13.844261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:33533 (local address 127.1.226.254:41127)
0928 04:43:13.844325 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:13.844328 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:13.844536 (+ 208us) server_negotiation.cc:407] Connection header received
0928 04:43:13.845122 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:13.845124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:13.845139 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:13.845223 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:13.846221 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.847080 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.848766 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.849065 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.849154 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:13.851958 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:13.851959 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:13.851960 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:13.851968 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:13.853204 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:13.853609 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:13.853610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:13.853610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:13.853671 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:13.853928 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:13.853929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:13.853930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:13.854056 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:13.854130 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:13.854404 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 04:43:13.854488 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 04:43:13.854768 4030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:13.844266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:41127 (local address 127.1.226.195:33533)
0928 04:43:13.844417 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:13.844455 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:13.845012 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:13.845310 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:13.845315 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:13.845723 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:13.846056 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:13.846062 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.847215 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.847217 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:13.848686 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:13.848689 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.849162 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.849163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:13.849309 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:13.850006 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:13.850017 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:13.851831 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:13.853301 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:13.853308 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:13.853310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:13.853532 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:13.853747 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:13.853748 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:13.853748 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:13.853864 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:13.854119 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:13.854125 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:13.854270 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 04:43:13.854371 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 04:43:13.854532 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260928 04:43:13.855259 4137 heartbeater.cc:347] Connected to a master server at 127.1.226.254:41127
I20260928 04:43:13.855365 4137 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:13.855540 4137 heartbeater.cc:511] Master 127.1.226.254:41127 requested a full tablet report, sending...
I20260928 04:43:13.855944 3653 ts_manager.cc:194] Registered new tserver with Master: 0ac7e274cd244f9c8ab79ab71210daf4 (127.1.226.195:43749)
I20260928 04:43:13.856352 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 4006
I20260928 04:43:13.856462 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance
I20260928 04:43:13.856468 3653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.195@KRBTEST.COM'} at 127.1.226.195:33533
I20260928 04:43:13.856669 1931 external_mini_cluster.cc:1428] Setting key cb6961f1c4a8f85428fef2b38cc7e2b7
I20260928 04:43:13.857630 1931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:43:13.863096 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:13.856492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:56333 (local address 127.1.226.254:41127)
0928 04:43:13.856812 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:13.856814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:13.856945 (+ 131us) server_negotiation.cc:407] Connection header received
0928 04:43:13.857028 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:13.857029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:13.857038 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:13.857082 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:13.857518 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.858235 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.859039 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.859354 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.859434 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:13.860626 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:13.860627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:13.860628 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:13.860634 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:13.861967 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:13.862314 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:13.862315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:13.862316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:13.862370 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:13.862559 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:13.862560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:13.862560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:13.862688 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:13.862767 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:13.862921 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 04:43:13.862998 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297}
I20260928 04:43:13.863108 4143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:13.856530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:41127 (local address 127.1.226.195:56333)
0928 04:43:13.856877 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:13.856888 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:13.856971 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:13.857083 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:13.857084 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:13.857373 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:13.857443 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:13.857445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.858341 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.858341 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:13.858948 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:13.858951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.859494 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.859495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:13.859549 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:13.860132 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:13.860138 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:13.860525 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:13.862050 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:13.862051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:13.862051 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:13.862230 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:13.862437 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:13.862438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:13.862438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:13.862498 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:13.862757 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:13.862758 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:13.862815 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 04:43:13.862896 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 04:43:13.863000 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":64,"threads_started":1}
Sep 28 04:43:13 dist-test-slave-j5c3 krb5kdc[3609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 04:43:13 dist-test-slave-j5c3 krb5kdc[3609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:13.881171 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:13.874242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48076 (local address 127.1.226.254:41127)
0928 04:43:13.874316 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:13.874318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:13.874326 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:43:13.874358 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:13.874358 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:13.874368 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:13.874428 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:13.875006 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.875529 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.876354 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.876558 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.876649 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:13.878906 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:13.878907 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:13.878908 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:13.878914 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:13.880136 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:13.880546 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:13.880546 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:13.880547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:13.880613 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:13.880772 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:13.880773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:13.880773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:13.880878 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:13.880953 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:13.881038 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 04:43:13.881103 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 04:43:13.882700 3653 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:43:13.884274 3653 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:43:13.891126 4157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:13.886818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56014 (local address 127.1.226.193:41701)
0928 04:43:13.886956 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:13.886962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:13.887301 (+ 339us) server_negotiation.cc:407] Connection header received
0928 04:43:13.887418 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:13.887421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:13.887438 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:13.887513 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:13.887764 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.888563 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.889964 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.890699 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.890787 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:13.890847 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:13.890988 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 04:43:13.891028 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":73,"threads_started":1}
I20260928 04:43:13.891141 4155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:13.886721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:41701 (local address 127.0.0.1:56014)
0928 04:43:13.887174 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:13.887213 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:13.887325 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:13.887599 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:13.887602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:13.887604 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:13.887675 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:13.887677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.888691 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.888692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:13.889880 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:13.889883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.890788 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.890789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:13.890881 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:13.890895 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 04:43:13.890964 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 04:43:13.891058 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":51,"threads_started":1}
I20260928 04:43:13.891150 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:13.886782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:43749 (local address 127.0.0.1:58604)
0928 04:43:13.887562 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:13.887571 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:13.887617 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:13.887831 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:13.887832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:13.887836 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:13.887878 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:13.887880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.889085 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.889086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:13.890084 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:13.890087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.890960 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.890961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:13.891013 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:13.891016 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:43:13.891061 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 04:43:13.891091 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":65,"threads_started":1}
I20260928 04:43:13.891512 4156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:13.886723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:43337 (local address 127.0.0.1:44952)
0928 04:43:13.887442 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:13.887450 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:13.887500 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:13.887696 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:13.887696 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:13.887698 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:13.887741 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:13.887742 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.889073 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.889074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:13.890265 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:13.890267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.891293 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.891294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:13.891355 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:13.891358 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:43:13.891405 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 04:43:13.891443 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"mutex_wait_us":118,"thread_start_us":89,"threads_started":1}
I20260928 04:43:13.891630 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:13.886983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58604 (local address 127.1.226.195:43749)
0928 04:43:13.887141 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:13.887146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:13.887630 (+ 484us) server_negotiation.cc:407] Connection header received
0928 04:43:13.887672 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:13.887674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:13.887690 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:13.887759 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:13.887951 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.888999 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.890179 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.890807 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.890873 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:13.890928 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:13.891332 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 04:43:13.891453 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260928 04:43:13.891867 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:13.886988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44952 (local address 127.1.226.194:43337)
0928 04:43:13.887142 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:13.887147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:13.887541 (+ 394us) server_negotiation.cc:407] Connection header received
0928 04:43:13.887590 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:13.887592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:13.887620 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:13.887705 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:13.887783 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.888986 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.890351 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:13.891108 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:13.891178 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:13.891258 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:13.891692 (+ 434us) server_negotiation.cc:299] Negotiation successful
0928 04:43:13.891736 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260928 04:43:13.892119 3791 tablet_service.cc:1467] Processing CreateTablet for tablet 5e583a044efe4f8993e43f0d7832bb5c (DEFAULT_TABLE table=test-table [id=6a4824196fbe4e2b811f62b7aefaec5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:13.892524 3791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e583a044efe4f8993e43f0d7832bb5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:13.892477 3931 tablet_service.cc:1467] Processing CreateTablet for tablet 5e583a044efe4f8993e43f0d7832bb5c (DEFAULT_TABLE table=test-table [id=6a4824196fbe4e2b811f62b7aefaec5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:13.892782 3931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e583a044efe4f8993e43f0d7832bb5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:13.895378 4160 tablet_bootstrap.cc:492] T 5e583a044efe4f8993e43f0d7832bb5c P 5c39d49bf2a04951a2094251619c837d: Bootstrap starting.
I20260928 04:43:13.895457 4161 tablet_bootstrap.cc:492] T 5e583a044efe4f8993e43f0d7832bb5c P f881bfbdccd44104bddfce14b5d0ba2a: Bootstrap starting.
I20260928 04:43:13.895905 4071 tablet_service.cc:1467] Processing CreateTablet for tablet 5e583a044efe4f8993e43f0d7832bb5c (DEFAULT_TABLE table=test-table [id=6a4824196fbe4e2b811f62b7aefaec5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:13.896046 4161 tablet_bootstrap.cc:654] T 5e583a044efe4f8993e43f0d7832bb5c P f881bfbdccd44104bddfce14b5d0ba2a: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:13.896092 4160 tablet_bootstrap.cc:654] T 5e583a044efe4f8993e43f0d7832bb5c P 5c39d49bf2a04951a2094251619c837d: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:13.896301 4071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e583a044efe4f8993e43f0d7832bb5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:13.896435 4160 log.cc:824] T 5e583a044efe4f8993e43f0d7832bb5c P 5c39d49bf2a04951a2094251619c837d: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:13.896432 4161 log.cc:824] T 5e583a044efe4f8993e43f0d7832bb5c P f881bfbdccd44104bddfce14b5d0ba2a: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:13.897174 4160 tablet_bootstrap.cc:492] T 5e583a044efe4f8993e43f0d7832bb5c P 5c39d49bf2a04951a2094251619c837d: No bootstrap required, opened a new log
I20260928 04:43:13.897240 4160 ts_tablet_manager.cc:1397] T 5e583a044efe4f8993e43f0d7832bb5c P 5c39d49bf2a04951a2094251619c837d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:43:13.897282 4161 tablet_bootstrap.cc:492] T 5e583a044efe4f8993e43f0d7832bb5c P f881bfbdccd44104bddfce14b5d0ba2a: No bootstrap required, opened a new log
I20260928 04:43:13.897351 4161 ts_tablet_manager.cc:1397] T 5e583a044efe4f8993e43f0d7832bb5c P f881bfbdccd44104bddfce14b5d0ba2a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 04:43:13.898921 4161 raft_consensus.cc:348] T 5e583a044efe4f8993e43f0d7832bb5c P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:13.899068 4161 raft_consensus.cc:374] T 5e583a044efe4f8993e43f0d7832bb5c P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:13.899108 4161 raft_consensus.cc:729] T 5e583a044efe4f8993e43f0d7832bb5c P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f881bfbdccd44104bddfce14b5d0ba2a, State: Initialized, Role: FOLLOWER
I20260928 04:43:13.899228 4161 consensus_queue.cc:260] T 5e583a044efe4f8993e43f0d7832bb5c P f881bfbdccd44104bddfce14b5d0ba2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:13.899228 4160 raft_consensus.cc:348] T 5e583a044efe4f8993e43f0d7832bb5c P 5c39d49bf2a04951a2094251619c837d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:13.899385 4160 raft_consensus.cc:374] T 5e583a044efe4f8993e43f0d7832bb5c P 5c39d49bf2a04951a2094251619c837d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:13.899438 4160 raft_consensus.cc:729] T 5e583a044efe4f8993e43f0d7832bb5c P 5c39d49bf2a04951a2094251619c837d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c39d49bf2a04951a2094251619c837d, State: Initialized, Role: FOLLOWER
I20260928 04:43:13.899492 4161 ts_tablet_manager.cc:1428] T 5e583a044efe4f8993e43f0d7832bb5c P f881bfbdccd44104bddfce14b5d0ba2a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 04:43:13.899554 4160 consensus_queue.cc:260] T 5e583a044efe4f8993e43f0d7832bb5c P 5c39d49bf2a04951a2094251619c837d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:13.899605 3997 heartbeater.cc:503] Master 127.1.226.254:41127 was elected leader, sending a full tablet report...
I20260928 04:43:13.899643 4164 tablet_bootstrap.cc:492] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4: Bootstrap starting.
I20260928 04:43:13.899812 4160 ts_tablet_manager.cc:1428] T 5e583a044efe4f8993e43f0d7832bb5c P 5c39d49bf2a04951a2094251619c837d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:43:13.899930 3857 heartbeater.cc:503] Master 127.1.226.254:41127 was elected leader, sending a full tablet report...
I20260928 04:43:13.900611 4164 tablet_bootstrap.cc:654] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:13.900914 4164 log.cc:824] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:13.901665 4164 tablet_bootstrap.cc:492] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4: No bootstrap required, opened a new log
I20260928 04:43:13.901736 4164 ts_tablet_manager.cc:1397] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 04:43:13.903010 4164 raft_consensus.cc:348] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:13.903144 4164 raft_consensus.cc:374] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:13.903187 4164 raft_consensus.cc:729] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac7e274cd244f9c8ab79ab71210daf4, State: Initialized, Role: FOLLOWER
I20260928 04:43:13.903302 4164 consensus_queue.cc:260] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:13.903541 4164 ts_tablet_manager.cc:1428] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 04:43:13.903566 4137 heartbeater.cc:503] Master 127.1.226.254:41127 was elected leader, sending a full tablet report...
W20260928 04:43:13.926281 3998 tablet.cc:2369] T 5e583a044efe4f8993e43f0d7832bb5c P f881bfbdccd44104bddfce14b5d0ba2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:43:14.010198 3858 tablet.cc:2369] T 5e583a044efe4f8993e43f0d7832bb5c P 5c39d49bf2a04951a2094251619c837d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:43:14.101838 4138 tablet.cc:2369] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:43:14.266491 4168 raft_consensus.cc:482] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:43:14.266616 4168 raft_consensus.cc:504] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:14.266968 4168 leader_election.cc:288] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f881bfbdccd44104bddfce14b5d0ba2a (127.1.226.194:43337), 5c39d49bf2a04951a2094251619c837d (127.1.226.193:41701)
I20260928 04:43:14.269856 4157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:14.267389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:48927 (local address 127.1.226.193:41701)
0928 04:43:14.267442 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:14.267444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:14.267453 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:43:14.267527 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:14.267528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:14.267536 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:14.267595 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:14.267784 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.268263 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.269153 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.269521 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.269601 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:14.269611 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:14.269752 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 04:43:14.269791 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 04:43:14.269866 4169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:14.267239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:41701 (local address 127.1.226.195:48927)
0928 04:43:14.267406 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:14.267416 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:14.267463 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:14.267610 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:14.267611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:14.267613 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:14.267684 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:14.267686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.268353 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.268354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:14.269069 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:14.269070 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.269599 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.269600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:14.269670 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:14.269674 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:43:14.269746 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 04:43:14.269782 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":111,"threads_started":1}
I20260928 04:43:14.269917 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:14.267241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:53373 (local address 127.1.226.194:43337)
0928 04:43:14.267303 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:14.267306 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:14.267320 (+ 14us) server_negotiation.cc:407] Connection header received
0928 04:43:14.267420 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:14.267421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:14.267435 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:14.267528 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:14.267682 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.268218 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.269176 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.269550 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.269646 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:14.269654 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:14.269805 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 04:43:14.269844 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 04:43:14.269951 4030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:14.267185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:43337 (local address 127.1.226.195:53373)
0928 04:43:14.267256 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:14.267269 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:14.267314 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:14.267530 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:14.267531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:14.267533 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:14.267609 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:14.267611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.268303 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.268304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:14.269065 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:14.269067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.269646 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.269647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:14.269732 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:14.269735 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:43:14.269798 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 04:43:14.269844 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 04:43:14.270234 3811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e583a044efe4f8993e43f0d7832bb5c" candidate_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c39d49bf2a04951a2094251619c837d" is_pre_election: true
I20260928 04:43:14.270237 3950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e583a044efe4f8993e43f0d7832bb5c" candidate_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" is_pre_election: true
I20260928 04:43:14.270392 3950 raft_consensus.cc:2454] T 5e583a044efe4f8993e43f0d7832bb5c P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ac7e274cd244f9c8ab79ab71210daf4 in term 0.
I20260928 04:43:14.270392 3811 raft_consensus.cc:2454] T 5e583a044efe4f8993e43f0d7832bb5c P 5c39d49bf2a04951a2094251619c837d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ac7e274cd244f9c8ab79ab71210daf4 in term 0.
I20260928 04:43:14.270606 4024 leader_election.cc:302] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ac7e274cd244f9c8ab79ab71210daf4, 5c39d49bf2a04951a2094251619c837d; no voters:
I20260928 04:43:14.270766 4168 raft_consensus.cc:2781] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:43:14.270829 4168 raft_consensus.cc:482] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:43:14.270865 4168 raft_consensus.cc:3037] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:14.271780 4168 raft_consensus.cc:504] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:14.271926 4168 leader_election.cc:288] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4 [CANDIDATE]: Term 1 election: Requested vote from peers f881bfbdccd44104bddfce14b5d0ba2a (127.1.226.194:43337), 5c39d49bf2a04951a2094251619c837d (127.1.226.193:41701)
I20260928 04:43:14.272092 3950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e583a044efe4f8993e43f0d7832bb5c" candidate_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f881bfbdccd44104bddfce14b5d0ba2a"
I20260928 04:43:14.272090 3811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e583a044efe4f8993e43f0d7832bb5c" candidate_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c39d49bf2a04951a2094251619c837d"
I20260928 04:43:14.272193 3950 raft_consensus.cc:3037] T 5e583a044efe4f8993e43f0d7832bb5c P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:14.272193 3811 raft_consensus.cc:3037] T 5e583a044efe4f8993e43f0d7832bb5c P 5c39d49bf2a04951a2094251619c837d [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:14.273047 3950 raft_consensus.cc:2454] T 5e583a044efe4f8993e43f0d7832bb5c P f881bfbdccd44104bddfce14b5d0ba2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ac7e274cd244f9c8ab79ab71210daf4 in term 1.
I20260928 04:43:14.273047 3811 raft_consensus.cc:2454] T 5e583a044efe4f8993e43f0d7832bb5c P 5c39d49bf2a04951a2094251619c837d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ac7e274cd244f9c8ab79ab71210daf4 in term 1.
I20260928 04:43:14.273267 4022 leader_election.cc:302] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ac7e274cd244f9c8ab79ab71210daf4, f881bfbdccd44104bddfce14b5d0ba2a; no voters:
I20260928 04:43:14.273357 4168 raft_consensus.cc:2781] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:14.273514 4168 raft_consensus.cc:686] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4 [term 1 LEADER]: Becoming Leader. State: Replica: 0ac7e274cd244f9c8ab79ab71210daf4, State: Running, Role: LEADER
I20260928 04:43:14.273617 4168 consensus_queue.cc:237] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:14.274456 3653 catalog_manager.cc:5495] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ac7e274cd244f9c8ab79ab71210daf4 (127.1.226.195). New cstate: current_term: 1 leader_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } health_report { overall_health: HEALTHY } } }
Sep 28 04:43:14 dist-test-slave-j5c3 krb5kdc[3609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570593, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:14.356009 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:14.348953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48080 (local address 127.1.226.254:41127)
0928 04:43:14.349035 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:14.349037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:14.349161 (+ 124us) server_negotiation.cc:407] Connection header received
0928 04:43:14.349240 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:14.349241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:14.349253 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:14.349322 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:14.349795 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.350280 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.351004 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.351182 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.351259 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:14.353372 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:14.353373 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:14.353374 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:14.353381 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:14.354703 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:14.355097 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:14.355098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:14.355098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:14.355295 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:14.355479 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:14.355480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:14.355480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:14.355595 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:14.355668 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:14.355808 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 04:43:14.355897 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 04:43:14.356099 4177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:14.348896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:41127 (local address 127.0.0.1:48080)
0928 04:43:14.349079 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:14.349096 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:14.349182 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:14.349326 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:14.349327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:14.349660 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:14.349711 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:14.349714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.350373 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.350374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:14.350924 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:14.350925 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.351257 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.351258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:14.351311 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:14.351861 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:14.351865 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:14.353275 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:14.354818 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:14.354825 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:14.354827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:14.355044 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:14.355356 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:14.355356 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:14.355357 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:14.355410 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:14.355658 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:14.355666 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:14.355711 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 04:43:14.355786 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 04:43:14.355959 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":44,"thread_start_us":100,"threads_started":1}
I20260928 04:43:14.358053 3653 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:48080:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 04:43:14.358318 3653 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:43:14.359916 3791 tablet_service.cc:1467] Processing CreateTablet for tablet 6f178ed0d7b646c98f9ad5544f12dca6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d80f44ec4e5458dbb52f979b1fab334]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 04:43:14.359934 4071 tablet_service.cc:1467] Processing CreateTablet for tablet 6f178ed0d7b646c98f9ad5544f12dca6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d80f44ec4e5458dbb52f979b1fab334]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 04:43:14.360095 4071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f178ed0d7b646c98f9ad5544f12dca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:14.360095 3791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f178ed0d7b646c98f9ad5544f12dca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:14.361219 4164 tablet_bootstrap.cc:492] T 6f178ed0d7b646c98f9ad5544f12dca6 P 0ac7e274cd244f9c8ab79ab71210daf4: Bootstrap starting.
I20260928 04:43:14.361349 3931 tablet_service.cc:1467] Processing CreateTablet for tablet 6f178ed0d7b646c98f9ad5544f12dca6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d80f44ec4e5458dbb52f979b1fab334]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 04:43:14.361409 4160 tablet_bootstrap.cc:492] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d: Bootstrap starting.
I20260928 04:43:14.361497 3931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f178ed0d7b646c98f9ad5544f12dca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:14.361650 4164 tablet_bootstrap.cc:654] T 6f178ed0d7b646c98f9ad5544f12dca6 P 0ac7e274cd244f9c8ab79ab71210daf4: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:14.361763 4160 tablet_bootstrap.cc:654] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:14.362287 4164 tablet_bootstrap.cc:492] T 6f178ed0d7b646c98f9ad5544f12dca6 P 0ac7e274cd244f9c8ab79ab71210daf4: No bootstrap required, opened a new log
I20260928 04:43:14.362344 4164 ts_tablet_manager.cc:1397] T 6f178ed0d7b646c98f9ad5544f12dca6 P 0ac7e274cd244f9c8ab79ab71210daf4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 04:43:14.362493 4164 raft_consensus.cc:348] T 6f178ed0d7b646c98f9ad5544f12dca6 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:14.362573 4164 raft_consensus.cc:374] T 6f178ed0d7b646c98f9ad5544f12dca6 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:14.362593 4164 raft_consensus.cc:729] T 6f178ed0d7b646c98f9ad5544f12dca6 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac7e274cd244f9c8ab79ab71210daf4, State: Initialized, Role: FOLLOWER
I20260928 04:43:14.362633 4164 consensus_queue.cc:260] T 6f178ed0d7b646c98f9ad5544f12dca6 P 0ac7e274cd244f9c8ab79ab71210daf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:14.362730 4164 ts_tablet_manager.cc:1428] T 6f178ed0d7b646c98f9ad5544f12dca6 P 0ac7e274cd244f9c8ab79ab71210daf4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:14.363091 4161 tablet_bootstrap.cc:492] T 6f178ed0d7b646c98f9ad5544f12dca6 P f881bfbdccd44104bddfce14b5d0ba2a: Bootstrap starting.
I20260928 04:43:14.363152 4168 tablet_replica.cc:439] T 6f178ed0d7b646c98f9ad5544f12dca6 P 0ac7e274cd244f9c8ab79ab71210daf4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } } }
I20260928 04:43:14.363232 4168 tablet_replica.cc:442] T 6f178ed0d7b646c98f9ad5544f12dca6 P 0ac7e274cd244f9c8ab79ab71210daf4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:43:14.363478 4161 tablet_bootstrap.cc:654] T 6f178ed0d7b646c98f9ad5544f12dca6 P f881bfbdccd44104bddfce14b5d0ba2a: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:14.363797 4160 tablet_bootstrap.cc:492] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d: No bootstrap required, opened a new log
I20260928 04:43:14.363857 4160 ts_tablet_manager.cc:1397] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:43:14.364014 4160 raft_consensus.cc:348] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:14.364082 4160 raft_consensus.cc:374] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:14.364097 4160 raft_consensus.cc:729] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c39d49bf2a04951a2094251619c837d, State: Initialized, Role: FOLLOWER
I20260928 04:43:14.364071 4161 tablet_bootstrap.cc:492] T 6f178ed0d7b646c98f9ad5544f12dca6 P f881bfbdccd44104bddfce14b5d0ba2a: No bootstrap required, opened a new log
I20260928 04:43:14.364130 4161 ts_tablet_manager.cc:1397] T 6f178ed0d7b646c98f9ad5544f12dca6 P f881bfbdccd44104bddfce14b5d0ba2a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 04:43:14.364180 4160 consensus_queue.cc:260] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:14.364305 4160 ts_tablet_manager.cc:1428] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 04:43:14.364328 4161 raft_consensus.cc:348] T 6f178ed0d7b646c98f9ad5544f12dca6 P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:14.364346 4166 tablet_replica.cc:439] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } } }
I20260928 04:43:14.364413 4161 raft_consensus.cc:374] T 6f178ed0d7b646c98f9ad5544f12dca6 P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:14.364414 4166 tablet_replica.cc:442] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:43:14.364444 4161 raft_consensus.cc:729] T 6f178ed0d7b646c98f9ad5544f12dca6 P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f881bfbdccd44104bddfce14b5d0ba2a, State: Initialized, Role: FOLLOWER
I20260928 04:43:14.364495 4161 consensus_queue.cc:260] T 6f178ed0d7b646c98f9ad5544f12dca6 P f881bfbdccd44104bddfce14b5d0ba2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:14.364598 4161 ts_tablet_manager.cc:1428] T 6f178ed0d7b646c98f9ad5544f12dca6 P f881bfbdccd44104bddfce14b5d0ba2a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:14.364656 4165 tablet_replica.cc:439] T 6f178ed0d7b646c98f9ad5544f12dca6 P f881bfbdccd44104bddfce14b5d0ba2a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } } }
I20260928 04:43:14.364727 4165 tablet_replica.cc:442] T 6f178ed0d7b646c98f9ad5544f12dca6 P f881bfbdccd44104bddfce14b5d0ba2a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:43:14.582966 4166 raft_consensus.cc:482] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:43:14.583081 4166 raft_consensus.cc:504] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:14.583438 4166 leader_election.cc:288] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f881bfbdccd44104bddfce14b5d0ba2a (127.1.226.194:43337), 0ac7e274cd244f9c8ab79ab71210daf4 (127.1.226.195:43749)
I20260928 04:43:14.586957 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:14.583679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:46853 (local address 127.1.226.194:43337)
0928 04:43:14.583727 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:14.583729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:14.583867 (+ 138us) server_negotiation.cc:407] Connection header received
0928 04:43:14.583916 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:14.583917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:14.583927 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:14.584002 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:14.584205 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.584969 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.586243 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.586595 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.586678 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:14.586688 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:14.586833 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 04:43:14.586880 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 04:43:14.586982 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:14.583632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:43749 (local address 127.1.226.193:37793)
0928 04:43:14.583777 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:14.583788 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:14.583851 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:14.584021 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:14.584021 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:14.584022 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:14.584106 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:14.584108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.585113 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.585114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:14.586093 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:14.586096 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.586693 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.586694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:14.586774 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:14.586778 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:43:14.586851 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 04:43:14.586889 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":54,"thread_start_us":109,"threads_started":1}
I20260928 04:43:14.586987 4181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:14.583631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:43337 (local address 127.1.226.193:46853)
0928 04:43:14.583771 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:14.583783 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:14.583851 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:14.584019 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:14.584020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:14.584022 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:14.584107 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:14.584108 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.585070 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.585072 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:14.586150 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:14.586154 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.586688 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.586689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:14.586751 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:14.586755 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:43:14.586807 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 04:43:14.586863 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":76,"thread_start_us":61,"threads_started":1}
I20260928 04:43:14.586987 4183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:14.583717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:37793 (local address 127.1.226.195:43749)
0928 04:43:14.583821 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:14.583823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:14.583851 (+ 28us) server_negotiation.cc:407] Connection header received
0928 04:43:14.583916 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:14.583917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:14.583928 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:14.584002 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:14.584206 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.584968 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.586175 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.586597 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.586684 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:14.586693 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:14.586866 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 04:43:14.586911 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":42,"threads_started":1}
I20260928 04:43:14.587297 4165 raft_consensus.cc:482] T 6f178ed0d7b646c98f9ad5544f12dca6 P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:43:14.587368 4165 raft_consensus.cc:504] T 6f178ed0d7b646c98f9ad5544f12dca6 P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:14.587435 3950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f178ed0d7b646c98f9ad5544f12dca6" candidate_uuid: "5c39d49bf2a04951a2094251619c837d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" is_pre_election: true
I20260928 04:43:14.587497 3950 raft_consensus.cc:2454] T 6f178ed0d7b646c98f9ad5544f12dca6 P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c39d49bf2a04951a2094251619c837d in term 0.
I20260928 04:43:14.587545 4091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f178ed0d7b646c98f9ad5544f12dca6" candidate_uuid: "5c39d49bf2a04951a2094251619c837d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" is_pre_election: true
I20260928 04:43:14.587672 4091 raft_consensus.cc:2454] T 6f178ed0d7b646c98f9ad5544f12dca6 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c39d49bf2a04951a2094251619c837d in term 0.
I20260928 04:43:14.587679 3742 leader_election.cc:302] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c39d49bf2a04951a2094251619c837d, f881bfbdccd44104bddfce14b5d0ba2a; no voters:
I20260928 04:43:14.587735 4165 leader_election.cc:288] T 6f178ed0d7b646c98f9ad5544f12dca6 P f881bfbdccd44104bddfce14b5d0ba2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c39d49bf2a04951a2094251619c837d (127.1.226.193:41701), 0ac7e274cd244f9c8ab79ab71210daf4 (127.1.226.195:43749)
I20260928 04:43:14.587816 4166 raft_consensus.cc:2781] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:43:14.587852 4166 raft_consensus.cc:482] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:43:14.587878 4166 raft_consensus.cc:3037] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:14.588909 4166 raft_consensus.cc:504] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:14.589046 4166 leader_election.cc:288] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d [CANDIDATE]: Term 1 election: Requested vote from peers f881bfbdccd44104bddfce14b5d0ba2a (127.1.226.194:43337), 0ac7e274cd244f9c8ab79ab71210daf4 (127.1.226.195:43749)
I20260928 04:43:14.589211 4091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f178ed0d7b646c98f9ad5544f12dca6" candidate_uuid: "5c39d49bf2a04951a2094251619c837d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac7e274cd244f9c8ab79ab71210daf4"
I20260928 04:43:14.589283 4091 raft_consensus.cc:3037] T 6f178ed0d7b646c98f9ad5544f12dca6 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:14.589272 3950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f178ed0d7b646c98f9ad5544f12dca6" candidate_uuid: "5c39d49bf2a04951a2094251619c837d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f881bfbdccd44104bddfce14b5d0ba2a"
I20260928 04:43:14.589370 3950 raft_consensus.cc:3037] T 6f178ed0d7b646c98f9ad5544f12dca6 P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:14.590056 4091 raft_consensus.cc:2454] T 6f178ed0d7b646c98f9ad5544f12dca6 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c39d49bf2a04951a2094251619c837d in term 1.
I20260928 04:43:14.590212 3743 leader_election.cc:302] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ac7e274cd244f9c8ab79ab71210daf4, 5c39d49bf2a04951a2094251619c837d; no voters:
I20260928 04:43:14.590394 4166 raft_consensus.cc:2781] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:14.590445 3950 raft_consensus.cc:2454] T 6f178ed0d7b646c98f9ad5544f12dca6 P f881bfbdccd44104bddfce14b5d0ba2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c39d49bf2a04951a2094251619c837d in term 1.
I20260928 04:43:14.590505 4166 raft_consensus.cc:686] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d [term 1 LEADER]: Becoming Leader. State: Replica: 5c39d49bf2a04951a2094251619c837d, State: Running, Role: LEADER
I20260928 04:43:14.590593 4166 consensus_queue.cc:237] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:14.590999 4166 tablet_replica.cc:439] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d: TxnStatusTablet state changed. Reason: New leader 5c39d49bf2a04951a2094251619c837d. Latest consensus state: current_term: 1 leader_uuid: "5c39d49bf2a04951a2094251619c837d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } } }
I20260928 04:43:14.591073 4166 tablet_replica.cc:442] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d: This TxnStatusTablet replica's current role is: LEADER
I20260928 04:43:14.591354 4157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:14.588017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:34403 (local address 127.1.226.193:41701)
0928 04:43:14.588052 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:14.588054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:14.588213 (+ 159us) server_negotiation.cc:407] Connection header received
0928 04:43:14.588275 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:14.588276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:14.588286 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:14.588333 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:14.588465 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.588975 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.590443 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.590944 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.590996 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:14.591003 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:14.591265 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 04:43:14.591303 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 04:43:14.591532 4183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:14.588040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:47139 (local address 127.1.226.195:43749)
0928 04:43:14.588075 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:14.588077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:14.588212 (+ 135us) server_negotiation.cc:407] Connection header received
0928 04:43:14.588316 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:14.588317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:14.588326 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:14.588375 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:14.588582 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.589221 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.590582 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.591122 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.591172 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:14.591181 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:14.591435 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 04:43:14.591474 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 04:43:14.591557 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:14.587879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:41701 (local address 127.1.226.194:34403)
0928 04:43:14.588120 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:14.588137 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:14.588228 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:14.588341 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:14.588342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:14.588343 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:14.588402 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:14.588403 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.589079 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.589080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:14.590349 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:14.590351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.591096 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.591097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:14.591174 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:14.591178 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:43:14.591408 (+ 230us) client_negotiation.cc:240] Negotiation successful
0928 04:43:14.591474 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":66,"threads_started":1}
I20260928 04:43:14.591759 4184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:14.587911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:43749 (local address 127.1.226.194:47139)
0928 04:43:14.588126 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:14.588136 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:14.588237 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:14.588412 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:14.588412 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:14.588414 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:14.588503 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:14.588505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.589303 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.589305 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:14.590452 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:14.590455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.591243 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.591244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:14.591326 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:14.591329 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:43:14.591635 (+ 306us) client_negotiation.cc:240] Negotiation successful
0928 04:43:14.591687 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":49,"threads_started":1}
I20260928 04:43:14.591684 3653 catalog_manager.cc:5495] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c39d49bf2a04951a2094251619c837d (127.1.226.193). New cstate: current_term: 1 leader_uuid: "5c39d49bf2a04951a2094251619c837d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } health_report { overall_health: UNKNOWN } } }
I20260928 04:43:14.592003 3811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f178ed0d7b646c98f9ad5544f12dca6" candidate_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c39d49bf2a04951a2094251619c837d" is_pre_election: true
I20260928 04:43:14.592005 4091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f178ed0d7b646c98f9ad5544f12dca6" candidate_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" is_pre_election: true
I20260928 04:43:14.592098 4091 raft_consensus.cc:2379] T 6f178ed0d7b646c98f9ad5544f12dca6 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f881bfbdccd44104bddfce14b5d0ba2a in current term 1: Already voted for candidate 5c39d49bf2a04951a2094251619c837d in this term.
I20260928 04:43:14.592074 4188 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 04:43:14.592196 4188 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 04:43:14.592373 3884 leader_election.cc:302] T 6f178ed0d7b646c98f9ad5544f12dca6 P f881bfbdccd44104bddfce14b5d0ba2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f881bfbdccd44104bddfce14b5d0ba2a; no voters: 0ac7e274cd244f9c8ab79ab71210daf4, 5c39d49bf2a04951a2094251619c837d
I20260928 04:43:14.592682 4165 raft_consensus.cc:2726] T 6f178ed0d7b646c98f9ad5544f12dca6 P f881bfbdccd44104bddfce14b5d0ba2a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 04:43:14.646484 4177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:14.643750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:41701 (local address 127.0.0.1:56026)
0928 04:43:14.643848 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:14.643861 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:14.643912 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:14.644102 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:14.644103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:14.644104 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:43:14.644227 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:14.644230 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.644967 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.644969 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:14.645577 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:14.645579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.645939 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.645940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:14.645999 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:14.646023 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:43:14.646306 (+ 283us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:43:14.646315 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 04:43:14.646364 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 04:43:14.646416 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 04:43:14.646477 4157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:14.643818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56026 (local address 127.1.226.193:41701)
0928 04:43:14.643878 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:14.643881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:14.643892 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:43:14.643988 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:14.643989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:14.644011 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:14.644093 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:43:14.644298 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.644887 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.645668 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.645868 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.645941 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:14.646128 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:43:14.646239 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:43:14.646329 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:14.646370 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 04:43:14.646391 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 04:43:14.648240 4091 raft_consensus.cc:1261] T 6f178ed0d7b646c98f9ad5544f12dca6 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 1 FOLLOWER]: Refusing update from remote peer 5c39d49bf2a04951a2094251619c837d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:14.648277 3950 raft_consensus.cc:1261] T 6f178ed0d7b646c98f9ad5544f12dca6 P f881bfbdccd44104bddfce14b5d0ba2a [term 1 FOLLOWER]: Refusing update from remote peer 5c39d49bf2a04951a2094251619c837d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:14.648571 4186 consensus_queue.cc:1035] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:14.648721 4166 consensus_queue.cc:1035] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:14.649871 4168 tablet_replica.cc:439] T 6f178ed0d7b646c98f9ad5544f12dca6 P 0ac7e274cd244f9c8ab79ab71210daf4: TxnStatusTablet state changed. Reason: New leader 5c39d49bf2a04951a2094251619c837d. Latest consensus state: current_term: 1 leader_uuid: "5c39d49bf2a04951a2094251619c837d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } } }
I20260928 04:43:14.649971 4168 tablet_replica.cc:442] T 6f178ed0d7b646c98f9ad5544f12dca6 P 0ac7e274cd244f9c8ab79ab71210daf4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:43:14.649875 4165 tablet_replica.cc:439] T 6f178ed0d7b646c98f9ad5544f12dca6 P f881bfbdccd44104bddfce14b5d0ba2a: TxnStatusTablet state changed. Reason: New leader 5c39d49bf2a04951a2094251619c837d. Latest consensus state: current_term: 1 leader_uuid: "5c39d49bf2a04951a2094251619c837d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } } }
I20260928 04:43:14.650002 4165 tablet_replica.cc:442] T 6f178ed0d7b646c98f9ad5544f12dca6 P f881bfbdccd44104bddfce14b5d0ba2a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:43:14.651281 4186 tablet_replica.cc:439] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c39d49bf2a04951a2094251619c837d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } } }
I20260928 04:43:14.651384 4186 tablet_replica.cc:442] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d: This TxnStatusTablet replica's current role is: LEADER
I20260928 04:43:14.651437 4186 tablet_replica.cc:439] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c39d49bf2a04951a2094251619c837d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } } }
I20260928 04:43:14.651420 4168 tablet_replica.cc:439] T 6f178ed0d7b646c98f9ad5544f12dca6 P 0ac7e274cd244f9c8ab79ab71210daf4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c39d49bf2a04951a2094251619c837d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } } }
I20260928 04:43:14.651494 4186 tablet_replica.cc:442] T 6f178ed0d7b646c98f9ad5544f12dca6 P 5c39d49bf2a04951a2094251619c837d: This TxnStatusTablet replica's current role is: LEADER
I20260928 04:43:14.651500 4168 tablet_replica.cc:442] T 6f178ed0d7b646c98f9ad5544f12dca6 P 0ac7e274cd244f9c8ab79ab71210daf4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:43:14.651527 4193 mvcc.cc:205] Tried to move back new op lower bound from 7334177155676651520 to 7334177155443613696. Current Snapshot: MvccSnapshot[applied={T|T < 7334177155676651520}]
I20260928 04:43:14.652009 4165 tablet_replica.cc:439] T 6f178ed0d7b646c98f9ad5544f12dca6 P f881bfbdccd44104bddfce14b5d0ba2a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c39d49bf2a04951a2094251619c837d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } } }
I20260928 04:43:14.652103 4165 tablet_replica.cc:442] T 6f178ed0d7b646c98f9ad5544f12dca6 P f881bfbdccd44104bddfce14b5d0ba2a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:43:14.652697 4192 mvcc.cc:205] Tried to move back new op lower bound from 7334177155676651520 to 7334177155443613696. Current Snapshot: MvccSnapshot[applied={T|T < 7334177155676651520}]
I20260928 04:43:14.656272 4183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:14.653680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58608 (local address 127.1.226.195:43749)
0928 04:43:14.653722 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:14.653724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:14.653733 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:43:14.653789 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:14.653790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:14.653807 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:14.653875 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:43:14.654032 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.654512 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.655381 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.655582 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.655661 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:14.655848 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:43:14.655985 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:43:14.656059 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:14.656109 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 04:43:14.656137 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:43:14.661477 4203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:14.658813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:41701 (local address 127.1.226.195:56755)
0928 04:43:14.658994 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:14.659004 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:14.659055 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:14.659193 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:14.659193 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:14.659194 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:43:14.659243 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:14.659244 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.660027 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.660028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:14.660647 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:14.660649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.661099 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.661100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:14.661151 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:14.661162 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:43:14.661345 (+ 183us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:43:14.661347 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:43:14.661378 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 04:43:14.661411 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":79,"threads_started":1}
I20260928 04:43:14.661484 4157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:14.658885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:56755 (local address 127.1.226.193:41701)
0928 04:43:14.658935 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:14.658937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:14.659067 (+ 130us) server_negotiation.cc:407] Connection header received
0928 04:43:14.659100 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:14.659101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:14.659113 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:14.659192 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:43:14.659294 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.659946 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.660727 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.661011 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.661086 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:14.661247 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:43:14.661282 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:43:14.661341 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:14.661393 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 04:43:14.661412 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 04:43:14.663949 3950 raft_consensus.cc:1261] T 5e583a044efe4f8993e43f0d7832bb5c P f881bfbdccd44104bddfce14b5d0ba2a [term 1 FOLLOWER]: Refusing update from remote peer 0ac7e274cd244f9c8ab79ab71210daf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:14.664052 3811 raft_consensus.cc:1261] T 5e583a044efe4f8993e43f0d7832bb5c P 5c39d49bf2a04951a2094251619c837d [term 1 FOLLOWER]: Refusing update from remote peer 0ac7e274cd244f9c8ab79ab71210daf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:14.664330 4170 consensus_queue.cc:1035] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:14.664458 4168 consensus_queue.cc:1035] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:14.666165 4191 mvcc.cc:205] Tried to move back new op lower bound from 7334177155741626368 to 7334177154145226752. Current Snapshot: MvccSnapshot[applied={T|T < 7334177155741626368}]
I20260928 04:43:14.673594 4183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:14.670937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:33593 (local address 127.1.226.195:43749)
0928 04:43:14.670973 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:14.670974 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:14.671108 (+ 134us) server_negotiation.cc:407] Connection header received
0928 04:43:14.671174 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:14.671175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:14.671186 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:14.671240 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:43:14.671373 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.671850 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.672908 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.673107 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.673183 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:14.673356 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:43:14.673388 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:43:14.673440 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:14.673499 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 04:43:14.673520 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 04:43:14.673635 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:14.670896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:43749 (local address 127.1.226.193:33593)
0928 04:43:14.671040 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:14.671057 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:14.671125 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:14.671243 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:14.671244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:14.671245 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:43:14.671308 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:14.671309 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.671929 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.671930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:14.672821 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:14.672824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.673185 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.673186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:14.673250 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:14.673269 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:43:14.673442 (+ 173us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:43:14.673444 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:43:14.673495 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 04:43:14.673539 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260928 04:43:14.681674 3654 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:43:14.681775 3654 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:43:14.682474 3654 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6955b3f0c50e4c0cb1864be5c8d8eb41: Sending DeleteTablet for 3 replicas of tablet 5e583a044efe4f8993e43f0d7832bb5c
I20260928 04:43:14.682752 3931 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e583a044efe4f8993e43f0d7832bb5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:14 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:44952
I20260928 04:43:14.683066 3791 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e583a044efe4f8993e43f0d7832bb5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:14 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:56014
I20260928 04:43:14.683357 4071 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e583a044efe4f8993e43f0d7832bb5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:14 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:58604
I20260928 04:43:14.683367 3654 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:43:14.683506 3654 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:43:14.684710 4071 tablet_service.cc:1467] Processing CreateTablet for tablet f2531b6bf5024c3ab19e5b966a2c85f6 (DEFAULT_TABLE table=test-table [id=f80ce269b0c64b028690bd63a18d45e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:14.684710 3931 tablet_service.cc:1467] Processing CreateTablet for tablet f2531b6bf5024c3ab19e5b966a2c85f6 (DEFAULT_TABLE table=test-table [id=f80ce269b0c64b028690bd63a18d45e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:14.684837 3931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2531b6bf5024c3ab19e5b966a2c85f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:14.684837 4071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2531b6bf5024c3ab19e5b966a2c85f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:14.686178 4164 tablet_bootstrap.cc:492] T f2531b6bf5024c3ab19e5b966a2c85f6 P 0ac7e274cd244f9c8ab79ab71210daf4: Bootstrap starting.
I20260928 04:43:14.686178 4161 tablet_bootstrap.cc:492] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a: Bootstrap starting.
I20260928 04:43:14.686566 4164 tablet_bootstrap.cc:654] T f2531b6bf5024c3ab19e5b966a2c85f6 P 0ac7e274cd244f9c8ab79ab71210daf4: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:14.686566 4161 tablet_bootstrap.cc:654] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:14.687194 3791 tablet_service.cc:1467] Processing CreateTablet for tablet f2531b6bf5024c3ab19e5b966a2c85f6 (DEFAULT_TABLE table=test-table [id=f80ce269b0c64b028690bd63a18d45e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:14.687332 3791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2531b6bf5024c3ab19e5b966a2c85f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:14.688488 4160 tablet_bootstrap.cc:492] T f2531b6bf5024c3ab19e5b966a2c85f6 P 5c39d49bf2a04951a2094251619c837d: Bootstrap starting.
I20260928 04:43:14.688876 4160 tablet_bootstrap.cc:654] T f2531b6bf5024c3ab19e5b966a2c85f6 P 5c39d49bf2a04951a2094251619c837d: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:14.689477 4160 tablet_bootstrap.cc:492] T f2531b6bf5024c3ab19e5b966a2c85f6 P 5c39d49bf2a04951a2094251619c837d: No bootstrap required, opened a new log
I20260928 04:43:14.689527 4160 ts_tablet_manager.cc:1397] T f2531b6bf5024c3ab19e5b966a2c85f6 P 5c39d49bf2a04951a2094251619c837d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:14.689679 4160 raft_consensus.cc:348] T f2531b6bf5024c3ab19e5b966a2c85f6 P 5c39d49bf2a04951a2094251619c837d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:14.689757 4160 raft_consensus.cc:374] T f2531b6bf5024c3ab19e5b966a2c85f6 P 5c39d49bf2a04951a2094251619c837d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:14.689785 4160 raft_consensus.cc:729] T f2531b6bf5024c3ab19e5b966a2c85f6 P 5c39d49bf2a04951a2094251619c837d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c39d49bf2a04951a2094251619c837d, State: Initialized, Role: FOLLOWER
I20260928 04:43:14.689822 4160 consensus_queue.cc:260] T f2531b6bf5024c3ab19e5b966a2c85f6 P 5c39d49bf2a04951a2094251619c837d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:14.689920 4160 ts_tablet_manager.cc:1428] T f2531b6bf5024c3ab19e5b966a2c85f6 P 5c39d49bf2a04951a2094251619c837d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 04:43:14.692272 4213 tablet_replica.cc:330] T 5e583a044efe4f8993e43f0d7832bb5c P f881bfbdccd44104bddfce14b5d0ba2a: stopping tablet replica
I20260928 04:43:14.692358 4214 tablet_replica.cc:330] T 5e583a044efe4f8993e43f0d7832bb5c P 5c39d49bf2a04951a2094251619c837d: stopping tablet replica
I20260928 04:43:14.692417 4213 raft_consensus.cc:2229] T 5e583a044efe4f8993e43f0d7832bb5c P f881bfbdccd44104bddfce14b5d0ba2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:43:14.692498 4213 raft_consensus.cc:2258] T 5e583a044efe4f8993e43f0d7832bb5c P f881bfbdccd44104bddfce14b5d0ba2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:14.692502 4214 raft_consensus.cc:2229] T 5e583a044efe4f8993e43f0d7832bb5c P 5c39d49bf2a04951a2094251619c837d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:43:14.692569 4214 raft_consensus.cc:2258] T 5e583a044efe4f8993e43f0d7832bb5c P 5c39d49bf2a04951a2094251619c837d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:14.692912 4213 ts_tablet_manager.cc:1905] T 5e583a044efe4f8993e43f0d7832bb5c P f881bfbdccd44104bddfce14b5d0ba2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:14.693298 4214 ts_tablet_manager.cc:1905] T 5e583a044efe4f8993e43f0d7832bb5c P 5c39d49bf2a04951a2094251619c837d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:14.694738 4213 ts_tablet_manager.cc:1918] T 5e583a044efe4f8993e43f0d7832bb5c P f881bfbdccd44104bddfce14b5d0ba2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:43:14.694787 4214 ts_tablet_manager.cc:1918] T 5e583a044efe4f8993e43f0d7832bb5c P 5c39d49bf2a04951a2094251619c837d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:43:14.694859 4214 log.cc:1196] T 5e583a044efe4f8993e43f0d7832bb5c P 5c39d49bf2a04951a2094251619c837d: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-0/wal/wals/5e583a044efe4f8993e43f0d7832bb5c
I20260928 04:43:14.694985 4213 log.cc:1196] T 5e583a044efe4f8993e43f0d7832bb5c P f881bfbdccd44104bddfce14b5d0ba2a: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-1/wal/wals/5e583a044efe4f8993e43f0d7832bb5c
I20260928 04:43:14.695195 4214 ts_tablet_manager.cc:1939] T 5e583a044efe4f8993e43f0d7832bb5c P 5c39d49bf2a04951a2094251619c837d: Deleting consensus metadata
I20260928 04:43:14.695441 4213 ts_tablet_manager.cc:1939] T 5e583a044efe4f8993e43f0d7832bb5c P f881bfbdccd44104bddfce14b5d0ba2a: Deleting consensus metadata
I20260928 04:43:14.695976 3638 catalog_manager.cc:4841] TS f881bfbdccd44104bddfce14b5d0ba2a (127.1.226.194:43337): tablet 5e583a044efe4f8993e43f0d7832bb5c (table test-table [id=6a4824196fbe4e2b811f62b7aefaec5c]) successfully deleted
I20260928 04:43:14.696513 3640 catalog_manager.cc:4841] TS 5c39d49bf2a04951a2094251619c837d (127.1.226.193:41701): tablet 5e583a044efe4f8993e43f0d7832bb5c (table test-table [id=6a4824196fbe4e2b811f62b7aefaec5c]) successfully deleted
I20260928 04:43:14.696570 4164 tablet_bootstrap.cc:492] T f2531b6bf5024c3ab19e5b966a2c85f6 P 0ac7e274cd244f9c8ab79ab71210daf4: No bootstrap required, opened a new log
I20260928 04:43:14.696630 4164 ts_tablet_manager.cc:1397] T f2531b6bf5024c3ab19e5b966a2c85f6 P 0ac7e274cd244f9c8ab79ab71210daf4: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.001s
I20260928 04:43:14.696766 4215 tablet_replica.cc:330] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4: stopping tablet replica
I20260928 04:43:14.696815 4164 raft_consensus.cc:348] T f2531b6bf5024c3ab19e5b966a2c85f6 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:14.696873 4215 raft_consensus.cc:2229] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:43:14.696902 4164 raft_consensus.cc:374] T f2531b6bf5024c3ab19e5b966a2c85f6 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:14.696926 4164 raft_consensus.cc:729] T f2531b6bf5024c3ab19e5b966a2c85f6 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac7e274cd244f9c8ab79ab71210daf4, State: Initialized, Role: FOLLOWER
I20260928 04:43:14.696978 4164 consensus_queue.cc:260] T f2531b6bf5024c3ab19e5b966a2c85f6 P 0ac7e274cd244f9c8ab79ab71210daf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:14.697105 4164 ts_tablet_manager.cc:1428] T f2531b6bf5024c3ab19e5b966a2c85f6 P 0ac7e274cd244f9c8ab79ab71210daf4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:14.697376 4161 tablet_bootstrap.cc:492] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a: No bootstrap required, opened a new log
I20260928 04:43:14.697428 4161 ts_tablet_manager.cc:1397] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.001s
I20260928 04:43:14.697620 4161 raft_consensus.cc:348] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:14.697711 4161 raft_consensus.cc:374] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:14.697747 4161 raft_consensus.cc:729] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f881bfbdccd44104bddfce14b5d0ba2a, State: Initialized, Role: FOLLOWER
I20260928 04:43:14.697803 4161 consensus_queue.cc:260] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:14.697916 4161 ts_tablet_manager.cc:1428] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 04:43:14.696983 4215 raft_consensus.cc:2258] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:14.698693 4215 ts_tablet_manager.cc:1905] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:14.700619 4215 ts_tablet_manager.cc:1918] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:43:14.700721 4215 log.cc:1196] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-2/wal/wals/5e583a044efe4f8993e43f0d7832bb5c
I20260928 04:43:14.701004 4215 ts_tablet_manager.cc:1939] T 5e583a044efe4f8993e43f0d7832bb5c P 0ac7e274cd244f9c8ab79ab71210daf4: Deleting consensus metadata
I20260928 04:43:14.701457 3639 catalog_manager.cc:4841] TS 0ac7e274cd244f9c8ab79ab71210daf4 (127.1.226.195:43749): tablet 5e583a044efe4f8993e43f0d7832bb5c (table test-table [id=6a4824196fbe4e2b811f62b7aefaec5c]) successfully deleted
I20260928 04:43:14.736256 4165 raft_consensus.cc:482] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:43:14.736370 4165 raft_consensus.cc:504] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:14.736536 4165 leader_election.cc:288] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c39d49bf2a04951a2094251619c837d (127.1.226.193:41701), 0ac7e274cd244f9c8ab79ab71210daf4 (127.1.226.195:43749)
I20260928 04:43:14.736708 3811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2531b6bf5024c3ab19e5b966a2c85f6" candidate_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c39d49bf2a04951a2094251619c837d" is_pre_election: true
I20260928 04:43:14.736810 3811 raft_consensus.cc:2454] T f2531b6bf5024c3ab19e5b966a2c85f6 P 5c39d49bf2a04951a2094251619c837d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f881bfbdccd44104bddfce14b5d0ba2a in term 0.
I20260928 04:43:14.736841 4091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2531b6bf5024c3ab19e5b966a2c85f6" candidate_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" is_pre_election: true
I20260928 04:43:14.736922 4091 raft_consensus.cc:2454] T f2531b6bf5024c3ab19e5b966a2c85f6 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f881bfbdccd44104bddfce14b5d0ba2a in term 0.
I20260928 04:43:14.736964 3884 leader_election.cc:302] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c39d49bf2a04951a2094251619c837d, f881bfbdccd44104bddfce14b5d0ba2a; no voters:
I20260928 04:43:14.737044 4165 raft_consensus.cc:2781] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:43:14.737071 4165 raft_consensus.cc:482] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:43:14.737092 4165 raft_consensus.cc:3037] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:14.737907 4165 raft_consensus.cc:504] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:14.738047 4165 leader_election.cc:288] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a [CANDIDATE]: Term 1 election: Requested vote from peers 5c39d49bf2a04951a2094251619c837d (127.1.226.193:41701), 0ac7e274cd244f9c8ab79ab71210daf4 (127.1.226.195:43749)
I20260928 04:43:14.738217 4091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2531b6bf5024c3ab19e5b966a2c85f6" candidate_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac7e274cd244f9c8ab79ab71210daf4"
I20260928 04:43:14.738204 3811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2531b6bf5024c3ab19e5b966a2c85f6" candidate_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c39d49bf2a04951a2094251619c837d"
I20260928 04:43:14.738267 4091 raft_consensus.cc:3037] T f2531b6bf5024c3ab19e5b966a2c85f6 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:14.738268 3811 raft_consensus.cc:3037] T f2531b6bf5024c3ab19e5b966a2c85f6 P 5c39d49bf2a04951a2094251619c837d [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:14.739012 3811 raft_consensus.cc:2454] T f2531b6bf5024c3ab19e5b966a2c85f6 P 5c39d49bf2a04951a2094251619c837d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f881bfbdccd44104bddfce14b5d0ba2a in term 1.
I20260928 04:43:14.739012 4091 raft_consensus.cc:2454] T f2531b6bf5024c3ab19e5b966a2c85f6 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f881bfbdccd44104bddfce14b5d0ba2a in term 1.
I20260928 04:43:14.739181 3884 leader_election.cc:302] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c39d49bf2a04951a2094251619c837d, f881bfbdccd44104bddfce14b5d0ba2a; no voters:
I20260928 04:43:14.739260 4165 raft_consensus.cc:2781] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:14.739384 4165 raft_consensus.cc:686] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a [term 1 LEADER]: Becoming Leader. State: Replica: f881bfbdccd44104bddfce14b5d0ba2a, State: Running, Role: LEADER
I20260928 04:43:14.739461 4165 consensus_queue.cc:237] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:14.740216 3654 catalog_manager.cc:5495] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a reported cstate change: term changed from 0 to 1, leader changed from <none> to f881bfbdccd44104bddfce14b5d0ba2a (127.1.226.194). New cstate: current_term: 1 leader_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } health_report { overall_health: UNKNOWN } } }
I20260928 04:43:14.751531 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:14.748938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44962 (local address 127.1.226.194:43337)
0928 04:43:14.748996 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:14.748998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:14.749043 (+ 45us) server_negotiation.cc:407] Connection header received
0928 04:43:14.749124 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:14.749125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:14.749149 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:14.749236 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:43:14.749360 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.749838 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.750698 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.750924 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.750995 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:14.751153 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:43:14.751262 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:43:14.751351 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:14.751403 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 04:43:14.751428 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 04:43:14.757119 4157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:14.754045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:55805 (local address 127.1.226.193:41701)
0928 04:43:14.754144 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:14.754146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:14.754212 (+ 66us) server_negotiation.cc:407] Connection header received
0928 04:43:14.754290 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:14.754309 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:14.754320 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:14.754386 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:43:14.754526 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.755031 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.756349 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.756568 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.756643 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:14.756851 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:43:14.756887 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:43:14.756951 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:14.757013 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 04:43:14.757034 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 04:43:14.757148 4224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:14.753981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:41701 (local address 127.1.226.194:55805)
0928 04:43:14.754147 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:14.754157 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:14.754228 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:14.754387 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:14.754388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:14.754390 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:43:14.754458 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:14.754460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.755107 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.755109 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:14.756261 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:14.756265 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.756662 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.756663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:14.756730 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:14.756745 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:43:14.756953 (+ 208us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:43:14.756956 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:43:14.757008 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 04:43:14.757054 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
I20260928 04:43:14.759747 3811 raft_consensus.cc:1261] T f2531b6bf5024c3ab19e5b966a2c85f6 P 5c39d49bf2a04951a2094251619c837d [term 1 FOLLOWER]: Refusing update from remote peer f881bfbdccd44104bddfce14b5d0ba2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:14.759980 4091 raft_consensus.cc:1261] T f2531b6bf5024c3ab19e5b966a2c85f6 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 1 FOLLOWER]: Refusing update from remote peer f881bfbdccd44104bddfce14b5d0ba2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:14.759999 4219 consensus_queue.cc:1035] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:14.760211 4219 consensus_queue.cc:1035] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:14.767827 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:14.765105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:43337 (local address 127.1.226.193:56353)
0928 04:43:14.765188 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:14.765200 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:14.765239 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:14.765403 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:14.765404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:14.765406 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:43:14.765448 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:14.765450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.766144 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.766146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:14.767015 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:14.767018 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.767483 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.767484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:14.767531 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:14.767537 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:43:14.767699 (+ 162us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:43:14.767701 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:43:14.767733 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 04:43:14.767767 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 04:43:14.767830 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:14.765181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:56353 (local address 127.1.226.194:43337)
0928 04:43:14.765226 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:14.765228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:14.765237 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:43:14.765292 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:14.765293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:14.765303 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:14.765357 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:43:14.765506 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.766067 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.767115 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:14.767415 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:14.767477 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:14.767602 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:43:14.767636 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:43:14.767697 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:14.767736 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 04:43:14.767756 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 04:43:14.768275 4210 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 04:43:16.294974 3652 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:43:16.295094 3652 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:43:16.296053 3652 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6955b3f0c50e4c0cb1864be5c8d8eb41: Sending DeleteTablet for 3 replicas of tablet f2531b6bf5024c3ab19e5b966a2c85f6
I20260928 04:43:16.296344 3791 tablet_service.cc:1514] Processing DeleteTablet for tablet f2531b6bf5024c3ab19e5b966a2c85f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:16 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:56014
I20260928 04:43:16.296411 3931 tablet_service.cc:1514] Processing DeleteTablet for tablet f2531b6bf5024c3ab19e5b966a2c85f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:16 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:44952
I20260928 04:43:16.296494 4071 tablet_service.cc:1514] Processing DeleteTablet for tablet f2531b6bf5024c3ab19e5b966a2c85f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:16 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:58604
I20260928 04:43:16.296622 4251 tablet_replica.cc:330] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a: stopping tablet replica
I20260928 04:43:16.296686 4251 raft_consensus.cc:2229] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:43:16.296794 4251 raft_consensus.cc:2258] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:16.297286 4251 ts_tablet_manager.cc:1905] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:16.298978 4250 tablet_replica.cc:330] T f2531b6bf5024c3ab19e5b966a2c85f6 P 5c39d49bf2a04951a2094251619c837d: stopping tablet replica
I20260928 04:43:16.299052 4251 ts_tablet_manager.cc:1918] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 04:43:16.299089 4252 tablet_replica.cc:330] T f2531b6bf5024c3ab19e5b966a2c85f6 P 0ac7e274cd244f9c8ab79ab71210daf4: stopping tablet replica
I20260928 04:43:16.299142 4251 log.cc:1196] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-1/wal/wals/f2531b6bf5024c3ab19e5b966a2c85f6
I20260928 04:43:16.299149 4252 raft_consensus.cc:2229] T f2531b6bf5024c3ab19e5b966a2c85f6 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:43:16.299192 4252 raft_consensus.cc:2258] T f2531b6bf5024c3ab19e5b966a2c85f6 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:16.299281 4250 raft_consensus.cc:2229] T f2531b6bf5024c3ab19e5b966a2c85f6 P 5c39d49bf2a04951a2094251619c837d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:43:16.299342 4250 raft_consensus.cc:2258] T f2531b6bf5024c3ab19e5b966a2c85f6 P 5c39d49bf2a04951a2094251619c837d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:16.299474 4251 ts_tablet_manager.cc:1939] T f2531b6bf5024c3ab19e5b966a2c85f6 P f881bfbdccd44104bddfce14b5d0ba2a: Deleting consensus metadata
I20260928 04:43:16.299623 4252 ts_tablet_manager.cc:1905] T f2531b6bf5024c3ab19e5b966a2c85f6 P 0ac7e274cd244f9c8ab79ab71210daf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:16.299657 4250 ts_tablet_manager.cc:1905] T f2531b6bf5024c3ab19e5b966a2c85f6 P 5c39d49bf2a04951a2094251619c837d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:16.299937 3638 catalog_manager.cc:4841] TS f881bfbdccd44104bddfce14b5d0ba2a (127.1.226.194:43337): tablet f2531b6bf5024c3ab19e5b966a2c85f6 (table test-table [id=f80ce269b0c64b028690bd63a18d45e5]) successfully deleted
I20260928 04:43:16.301359 4250 ts_tablet_manager.cc:1918] T f2531b6bf5024c3ab19e5b966a2c85f6 P 5c39d49bf2a04951a2094251619c837d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 04:43:16.301442 4250 log.cc:1196] T f2531b6bf5024c3ab19e5b966a2c85f6 P 5c39d49bf2a04951a2094251619c837d: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-0/wal/wals/f2531b6bf5024c3ab19e5b966a2c85f6
I20260928 04:43:16.301617 4252 ts_tablet_manager.cc:1918] T f2531b6bf5024c3ab19e5b966a2c85f6 P 0ac7e274cd244f9c8ab79ab71210daf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 04:43:16.301692 4252 log.cc:1196] T f2531b6bf5024c3ab19e5b966a2c85f6 P 0ac7e274cd244f9c8ab79ab71210daf4: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-2/wal/wals/f2531b6bf5024c3ab19e5b966a2c85f6
I20260928 04:43:16.301712 4250 ts_tablet_manager.cc:1939] T f2531b6bf5024c3ab19e5b966a2c85f6 P 5c39d49bf2a04951a2094251619c837d: Deleting consensus metadata
I20260928 04:43:16.301921 4252 ts_tablet_manager.cc:1939] T f2531b6bf5024c3ab19e5b966a2c85f6 P 0ac7e274cd244f9c8ab79ab71210daf4: Deleting consensus metadata
I20260928 04:43:16.302107 3640 catalog_manager.cc:4841] TS 5c39d49bf2a04951a2094251619c837d (127.1.226.193:41701): tablet f2531b6bf5024c3ab19e5b966a2c85f6 (table test-table [id=f80ce269b0c64b028690bd63a18d45e5]) successfully deleted
I20260928 04:43:16.302227 3639 catalog_manager.cc:4841] TS 0ac7e274cd244f9c8ab79ab71210daf4 (127.1.226.195:43749): tablet f2531b6bf5024c3ab19e5b966a2c85f6 (table test-table [id=f80ce269b0c64b028690bd63a18d45e5]) successfully deleted
Sep 28 04:43:16 dist-test-slave-j5c3 krb5kdc[3609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570596, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 04:43:16 dist-test-slave-j5c3 krb5kdc[3609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570596, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:16.327890 4260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:16.313884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57362 (local address 127.1.226.254:41127)
0928 04:43:16.314046 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:16.314048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:16.314085 (+ 37us) server_negotiation.cc:407] Connection header received
0928 04:43:16.314162 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:16.314163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:16.314176 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:16.314260 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:16.314923 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:16.315429 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:16.316148 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:16.316394 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:16.316450 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:16.325065 (+ 8615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:16.325068 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:16.325068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:16.325076 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:16.326628 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:16.327045 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:16.327047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:16.327047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:16.327112 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:16.327332 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:16.327333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:16.327333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:16.327456 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:16.327567 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:16.327678 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 04:43:16.327767 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":109,"threads_started":1}
I20260928 04:43:16.329627 3652 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:43:16.329879 3652 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:43:16.331187 3791 tablet_service.cc:1467] Processing CreateTablet for tablet ed17ab9acf464d50979d3a27f772c317 (DEFAULT_TABLE table=test-table [id=820001ff371e4ffca7485870f1d48aff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:16.331199 4071 tablet_service.cc:1467] Processing CreateTablet for tablet ed17ab9acf464d50979d3a27f772c317 (DEFAULT_TABLE table=test-table [id=820001ff371e4ffca7485870f1d48aff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:16.331223 3931 tablet_service.cc:1467] Processing CreateTablet for tablet ed17ab9acf464d50979d3a27f772c317 (DEFAULT_TABLE table=test-table [id=820001ff371e4ffca7485870f1d48aff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:16.331331 3791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed17ab9acf464d50979d3a27f772c317. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:16.331331 4071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed17ab9acf464d50979d3a27f772c317. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:16.331362 3931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed17ab9acf464d50979d3a27f772c317. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:16.332655 4261 tablet_bootstrap.cc:492] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a: Bootstrap starting.
I20260928 04:43:16.333022 4261 tablet_bootstrap.cc:654] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:16.333313 4262 tablet_bootstrap.cc:492] T ed17ab9acf464d50979d3a27f772c317 P 5c39d49bf2a04951a2094251619c837d: Bootstrap starting.
I20260928 04:43:16.333595 4261 tablet_bootstrap.cc:492] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a: No bootstrap required, opened a new log
I20260928 04:43:16.333637 4261 ts_tablet_manager.cc:1397] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:16.333693 4262 tablet_bootstrap.cc:654] T ed17ab9acf464d50979d3a27f772c317 P 5c39d49bf2a04951a2094251619c837d: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:16.333755 4261 raft_consensus.cc:348] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:16.333832 4261 raft_consensus.cc:374] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:16.333858 4261 raft_consensus.cc:729] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f881bfbdccd44104bddfce14b5d0ba2a, State: Initialized, Role: FOLLOWER
I20260928 04:43:16.333928 4261 consensus_queue.cc:260] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:16.334017 4263 tablet_bootstrap.cc:492] T ed17ab9acf464d50979d3a27f772c317 P 0ac7e274cd244f9c8ab79ab71210daf4: Bootstrap starting.
I20260928 04:43:16.334031 4261 ts_tablet_manager.cc:1428] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:16.334383 4263 tablet_bootstrap.cc:654] T ed17ab9acf464d50979d3a27f772c317 P 0ac7e274cd244f9c8ab79ab71210daf4: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:16.334959 4263 tablet_bootstrap.cc:492] T ed17ab9acf464d50979d3a27f772c317 P 0ac7e274cd244f9c8ab79ab71210daf4: No bootstrap required, opened a new log
I20260928 04:43:16.335001 4263 ts_tablet_manager.cc:1397] T ed17ab9acf464d50979d3a27f772c317 P 0ac7e274cd244f9c8ab79ab71210daf4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:16.335023 4262 tablet_bootstrap.cc:492] T ed17ab9acf464d50979d3a27f772c317 P 5c39d49bf2a04951a2094251619c837d: No bootstrap required, opened a new log
I20260928 04:43:16.335078 4262 ts_tablet_manager.cc:1397] T ed17ab9acf464d50979d3a27f772c317 P 5c39d49bf2a04951a2094251619c837d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:43:16.335109 4263 raft_consensus.cc:348] T ed17ab9acf464d50979d3a27f772c317 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:16.335178 4263 raft_consensus.cc:374] T ed17ab9acf464d50979d3a27f772c317 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:16.335213 4263 raft_consensus.cc:729] T ed17ab9acf464d50979d3a27f772c317 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac7e274cd244f9c8ab79ab71210daf4, State: Initialized, Role: FOLLOWER
I20260928 04:43:16.335222 4262 raft_consensus.cc:348] T ed17ab9acf464d50979d3a27f772c317 P 5c39d49bf2a04951a2094251619c837d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:16.335294 4262 raft_consensus.cc:374] T ed17ab9acf464d50979d3a27f772c317 P 5c39d49bf2a04951a2094251619c837d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:16.335284 4263 consensus_queue.cc:260] T ed17ab9acf464d50979d3a27f772c317 P 0ac7e274cd244f9c8ab79ab71210daf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:16.335326 4262 raft_consensus.cc:729] T ed17ab9acf464d50979d3a27f772c317 P 5c39d49bf2a04951a2094251619c837d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c39d49bf2a04951a2094251619c837d, State: Initialized, Role: FOLLOWER
I20260928 04:43:16.335384 4262 consensus_queue.cc:260] T ed17ab9acf464d50979d3a27f772c317 P 5c39d49bf2a04951a2094251619c837d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:16.335443 4263 ts_tablet_manager.cc:1428] T ed17ab9acf464d50979d3a27f772c317 P 0ac7e274cd244f9c8ab79ab71210daf4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 04:43:16.335644 4262 ts_tablet_manager.cc:1428] T ed17ab9acf464d50979d3a27f772c317 P 5c39d49bf2a04951a2094251619c837d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:16.509788 4245 raft_consensus.cc:482] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:43:16.509886 4245 raft_consensus.cc:504] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:16.510069 4245 leader_election.cc:288] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c39d49bf2a04951a2094251619c837d (127.1.226.193:41701), 0ac7e274cd244f9c8ab79ab71210daf4 (127.1.226.195:43749)
I20260928 04:43:16.510334 4091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed17ab9acf464d50979d3a27f772c317" candidate_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" is_pre_election: true
I20260928 04:43:16.510303 3811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed17ab9acf464d50979d3a27f772c317" candidate_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c39d49bf2a04951a2094251619c837d" is_pre_election: true
I20260928 04:43:16.510435 3811 raft_consensus.cc:2454] T ed17ab9acf464d50979d3a27f772c317 P 5c39d49bf2a04951a2094251619c837d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f881bfbdccd44104bddfce14b5d0ba2a in term 0.
I20260928 04:43:16.510440 4091 raft_consensus.cc:2454] T ed17ab9acf464d50979d3a27f772c317 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f881bfbdccd44104bddfce14b5d0ba2a in term 0.
I20260928 04:43:16.510612 3884 leader_election.cc:302] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c39d49bf2a04951a2094251619c837d, f881bfbdccd44104bddfce14b5d0ba2a; no voters:
I20260928 04:43:16.510710 4245 raft_consensus.cc:2781] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:43:16.510744 4245 raft_consensus.cc:482] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:43:16.510775 4245 raft_consensus.cc:3037] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:16.511518 4245 raft_consensus.cc:504] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:16.511656 4245 leader_election.cc:288] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a [CANDIDATE]: Term 1 election: Requested vote from peers 5c39d49bf2a04951a2094251619c837d (127.1.226.193:41701), 0ac7e274cd244f9c8ab79ab71210daf4 (127.1.226.195:43749)
I20260928 04:43:16.511821 3811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed17ab9acf464d50979d3a27f772c317" candidate_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c39d49bf2a04951a2094251619c837d"
I20260928 04:43:16.511857 4091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed17ab9acf464d50979d3a27f772c317" candidate_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac7e274cd244f9c8ab79ab71210daf4"
I20260928 04:43:16.511931 4091 raft_consensus.cc:3037] T ed17ab9acf464d50979d3a27f772c317 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:16.511934 3811 raft_consensus.cc:3037] T ed17ab9acf464d50979d3a27f772c317 P 5c39d49bf2a04951a2094251619c837d [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:16.512677 4091 raft_consensus.cc:2454] T ed17ab9acf464d50979d3a27f772c317 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f881bfbdccd44104bddfce14b5d0ba2a in term 1.
I20260928 04:43:16.512810 3811 raft_consensus.cc:2454] T ed17ab9acf464d50979d3a27f772c317 P 5c39d49bf2a04951a2094251619c837d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f881bfbdccd44104bddfce14b5d0ba2a in term 1.
I20260928 04:43:16.512866 3883 leader_election.cc:302] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ac7e274cd244f9c8ab79ab71210daf4, f881bfbdccd44104bddfce14b5d0ba2a; no voters:
I20260928 04:43:16.512961 4245 raft_consensus.cc:2781] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:16.513041 4245 raft_consensus.cc:686] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a [term 1 LEADER]: Becoming Leader. State: Replica: f881bfbdccd44104bddfce14b5d0ba2a, State: Running, Role: LEADER
I20260928 04:43:16.513092 4245 consensus_queue.cc:237] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:16.513695 3654 catalog_manager.cc:5495] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a reported cstate change: term changed from 0 to 1, leader changed from <none> to f881bfbdccd44104bddfce14b5d0ba2a (127.1.226.194). New cstate: current_term: 1 leader_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } health_report { overall_health: UNKNOWN } } }
I20260928 04:43:16.565122 4270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:16.562689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54572 (local address 127.1.226.194:43337)
0928 04:43:16.562876 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:16.562878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:16.562884 (+ 6us) server_negotiation.cc:407] Connection header received
0928 04:43:16.562901 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:16.562902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:16.562911 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:16.562983 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:43:16.563096 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:16.563582 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:16.564379 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:16.564594 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:16.564677 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:16.564846 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:43:16.564880 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:43:16.564934 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:16.565003 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 04:43:16.565025 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":79,"threads_started":1}
I20260928 04:43:16.567188 4091 raft_consensus.cc:1261] T ed17ab9acf464d50979d3a27f772c317 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 1 FOLLOWER]: Refusing update from remote peer f881bfbdccd44104bddfce14b5d0ba2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:16.567216 3811 raft_consensus.cc:1261] T ed17ab9acf464d50979d3a27f772c317 P 5c39d49bf2a04951a2094251619c837d [term 1 FOLLOWER]: Refusing update from remote peer f881bfbdccd44104bddfce14b5d0ba2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:16.567449 4241 consensus_queue.cc:1035] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:16.567570 4245 consensus_queue.cc:1035] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:16.577091 3654 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:43:16.577183 3654 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:43:16.577736 3654 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6955b3f0c50e4c0cb1864be5c8d8eb41: Sending DeleteTablet for 3 replicas of tablet ed17ab9acf464d50979d3a27f772c317
I20260928 04:43:16.577982 3791 tablet_service.cc:1514] Processing DeleteTablet for tablet ed17ab9acf464d50979d3a27f772c317 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:16 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:56014
I20260928 04:43:16.578020 3931 tablet_service.cc:1514] Processing DeleteTablet for tablet ed17ab9acf464d50979d3a27f772c317 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:16 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:44952
I20260928 04:43:16.578055 4071 tablet_service.cc:1514] Processing DeleteTablet for tablet ed17ab9acf464d50979d3a27f772c317 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:16 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:58604
I20260928 04:43:16.578082 4250 tablet_replica.cc:330] T ed17ab9acf464d50979d3a27f772c317 P 5c39d49bf2a04951a2094251619c837d: stopping tablet replica
I20260928 04:43:16.578135 4252 tablet_replica.cc:330] T ed17ab9acf464d50979d3a27f772c317 P 0ac7e274cd244f9c8ab79ab71210daf4: stopping tablet replica
I20260928 04:43:16.578140 4250 raft_consensus.cc:2229] T ed17ab9acf464d50979d3a27f772c317 P 5c39d49bf2a04951a2094251619c837d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:43:16.578177 4250 raft_consensus.cc:2258] T ed17ab9acf464d50979d3a27f772c317 P 5c39d49bf2a04951a2094251619c837d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:16.578224 4252 raft_consensus.cc:2229] T ed17ab9acf464d50979d3a27f772c317 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:43:16.578245 4251 tablet_replica.cc:330] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a: stopping tablet replica
I20260928 04:43:16.578271 4252 raft_consensus.cc:2258] T ed17ab9acf464d50979d3a27f772c317 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:16.578305 4251 raft_consensus.cc:2229] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:43:16.578361 4251 raft_consensus.cc:2258] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:16.578382 3654 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:43:16.578533 3654 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:43:16.578704 4252 ts_tablet_manager.cc:1905] T ed17ab9acf464d50979d3a27f772c317 P 0ac7e274cd244f9c8ab79ab71210daf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:16.578941 4250 ts_tablet_manager.cc:1905] T ed17ab9acf464d50979d3a27f772c317 P 5c39d49bf2a04951a2094251619c837d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:16.580019 4251 ts_tablet_manager.cc:1905] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:16.580330 4250 ts_tablet_manager.cc:1918] T ed17ab9acf464d50979d3a27f772c317 P 5c39d49bf2a04951a2094251619c837d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:43:16.580387 4250 log.cc:1196] T ed17ab9acf464d50979d3a27f772c317 P 5c39d49bf2a04951a2094251619c837d: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-0/wal/wals/ed17ab9acf464d50979d3a27f772c317
I20260928 04:43:16.580605 4250 ts_tablet_manager.cc:1939] T ed17ab9acf464d50979d3a27f772c317 P 5c39d49bf2a04951a2094251619c837d: Deleting consensus metadata
I20260928 04:43:16.580899 3640 catalog_manager.cc:4841] TS 5c39d49bf2a04951a2094251619c837d (127.1.226.193:41701): tablet ed17ab9acf464d50979d3a27f772c317 (table test-table [id=820001ff371e4ffca7485870f1d48aff]) successfully deleted
I20260928 04:43:16.581355 3931 tablet_service.cc:1467] Processing CreateTablet for tablet bceda5b45c384571a84d046af258c194 (DEFAULT_TABLE table=test-table [id=f907683cc4fc43459e3fe0eb7354b5c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:16.581369 4071 tablet_service.cc:1467] Processing CreateTablet for tablet bceda5b45c384571a84d046af258c194 (DEFAULT_TABLE table=test-table [id=f907683cc4fc43459e3fe0eb7354b5c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:16.581460 3931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bceda5b45c384571a84d046af258c194. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:16.581452 3791 tablet_service.cc:1467] Processing CreateTablet for tablet bceda5b45c384571a84d046af258c194 (DEFAULT_TABLE table=test-table [id=f907683cc4fc43459e3fe0eb7354b5c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:16.581557 3791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bceda5b45c384571a84d046af258c194. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:16.581594 4251 ts_tablet_manager.cc:1918] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:43:16.581641 4251 log.cc:1196] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-1/wal/wals/ed17ab9acf464d50979d3a27f772c317
I20260928 04:43:16.581854 4251 ts_tablet_manager.cc:1939] T ed17ab9acf464d50979d3a27f772c317 P f881bfbdccd44104bddfce14b5d0ba2a: Deleting consensus metadata
I20260928 04:43:16.582018 4252 ts_tablet_manager.cc:1918] T ed17ab9acf464d50979d3a27f772c317 P 0ac7e274cd244f9c8ab79ab71210daf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:43:16.582033 4071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bceda5b45c384571a84d046af258c194. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:16.582062 4252 log.cc:1196] T ed17ab9acf464d50979d3a27f772c317 P 0ac7e274cd244f9c8ab79ab71210daf4: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-2/wal/wals/ed17ab9acf464d50979d3a27f772c317
I20260928 04:43:16.582267 4252 ts_tablet_manager.cc:1939] T ed17ab9acf464d50979d3a27f772c317 P 0ac7e274cd244f9c8ab79ab71210daf4: Deleting consensus metadata
I20260928 04:43:16.582557 4261 tablet_bootstrap.cc:492] T bceda5b45c384571a84d046af258c194 P f881bfbdccd44104bddfce14b5d0ba2a: Bootstrap starting.
I20260928 04:43:16.582856 4262 tablet_bootstrap.cc:492] T bceda5b45c384571a84d046af258c194 P 5c39d49bf2a04951a2094251619c837d: Bootstrap starting.
I20260928 04:43:16.582894 4261 tablet_bootstrap.cc:654] T bceda5b45c384571a84d046af258c194 P f881bfbdccd44104bddfce14b5d0ba2a: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:16.583135 4262 tablet_bootstrap.cc:654] T bceda5b45c384571a84d046af258c194 P 5c39d49bf2a04951a2094251619c837d: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:16.583191 3639 catalog_manager.cc:4841] TS 0ac7e274cd244f9c8ab79ab71210daf4 (127.1.226.195:43749): tablet ed17ab9acf464d50979d3a27f772c317 (table test-table [id=820001ff371e4ffca7485870f1d48aff]) successfully deleted
I20260928 04:43:16.583273 4263 tablet_bootstrap.cc:492] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4: Bootstrap starting.
I20260928 04:43:16.583544 4263 tablet_bootstrap.cc:654] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:16.583725 4262 tablet_bootstrap.cc:492] T bceda5b45c384571a84d046af258c194 P 5c39d49bf2a04951a2094251619c837d: No bootstrap required, opened a new log
I20260928 04:43:16.583772 4262 ts_tablet_manager.cc:1397] T bceda5b45c384571a84d046af258c194 P 5c39d49bf2a04951a2094251619c837d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:16.583882 4261 tablet_bootstrap.cc:492] T bceda5b45c384571a84d046af258c194 P f881bfbdccd44104bddfce14b5d0ba2a: No bootstrap required, opened a new log
I20260928 04:43:16.583927 4261 ts_tablet_manager.cc:1397] T bceda5b45c384571a84d046af258c194 P f881bfbdccd44104bddfce14b5d0ba2a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:16.583920 4262 raft_consensus.cc:348] T bceda5b45c384571a84d046af258c194 P 5c39d49bf2a04951a2094251619c837d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:16.583971 3638 catalog_manager.cc:4841] TS f881bfbdccd44104bddfce14b5d0ba2a (127.1.226.194:43337): tablet ed17ab9acf464d50979d3a27f772c317 (table test-table [id=820001ff371e4ffca7485870f1d48aff]) successfully deleted
I20260928 04:43:16.583994 4262 raft_consensus.cc:374] T bceda5b45c384571a84d046af258c194 P 5c39d49bf2a04951a2094251619c837d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:16.584016 4262 raft_consensus.cc:729] T bceda5b45c384571a84d046af258c194 P 5c39d49bf2a04951a2094251619c837d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c39d49bf2a04951a2094251619c837d, State: Initialized, Role: FOLLOWER
I20260928 04:43:16.584055 4262 consensus_queue.cc:260] T bceda5b45c384571a84d046af258c194 P 5c39d49bf2a04951a2094251619c837d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:16.584033 4261 raft_consensus.cc:348] T bceda5b45c384571a84d046af258c194 P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:16.584102 4261 raft_consensus.cc:374] T bceda5b45c384571a84d046af258c194 P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:16.584123 4261 raft_consensus.cc:729] T bceda5b45c384571a84d046af258c194 P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f881bfbdccd44104bddfce14b5d0ba2a, State: Initialized, Role: FOLLOWER
I20260928 04:43:16.584120 4263 tablet_bootstrap.cc:492] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4: No bootstrap required, opened a new log
I20260928 04:43:16.584187 4263 ts_tablet_manager.cc:1397] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:16.584195 4262 ts_tablet_manager.cc:1428] T bceda5b45c384571a84d046af258c194 P 5c39d49bf2a04951a2094251619c837d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 04:43:16.584198 4261 consensus_queue.cc:260] T bceda5b45c384571a84d046af258c194 P f881bfbdccd44104bddfce14b5d0ba2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:16.584295 4261 ts_tablet_manager.cc:1428] T bceda5b45c384571a84d046af258c194 P f881bfbdccd44104bddfce14b5d0ba2a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:16.584285 4263 raft_consensus.cc:348] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:16.584352 4263 raft_consensus.cc:374] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:16.584376 4263 raft_consensus.cc:729] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac7e274cd244f9c8ab79ab71210daf4, State: Initialized, Role: FOLLOWER
I20260928 04:43:16.584429 4263 consensus_queue.cc:260] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:16.584513 4263 ts_tablet_manager.cc:1428] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:16.648872 4267 raft_consensus.cc:482] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:43:16.648986 4267 raft_consensus.cc:504] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:16.649160 4267 leader_election.cc:288] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c39d49bf2a04951a2094251619c837d (127.1.226.193:41701), f881bfbdccd44104bddfce14b5d0ba2a (127.1.226.194:43337)
I20260928 04:43:16.649410 3950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bceda5b45c384571a84d046af258c194" candidate_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" is_pre_election: true
I20260928 04:43:16.649415 3811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bceda5b45c384571a84d046af258c194" candidate_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c39d49bf2a04951a2094251619c837d" is_pre_election: true
I20260928 04:43:16.649521 3811 raft_consensus.cc:2454] T bceda5b45c384571a84d046af258c194 P 5c39d49bf2a04951a2094251619c837d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ac7e274cd244f9c8ab79ab71210daf4 in term 0.
I20260928 04:43:16.649525 3950 raft_consensus.cc:2454] T bceda5b45c384571a84d046af258c194 P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ac7e274cd244f9c8ab79ab71210daf4 in term 0.
I20260928 04:43:16.649686 4024 leader_election.cc:302] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ac7e274cd244f9c8ab79ab71210daf4, 5c39d49bf2a04951a2094251619c837d; no voters:
I20260928 04:43:16.649803 4267 raft_consensus.cc:2781] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:43:16.649876 4267 raft_consensus.cc:482] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:43:16.649909 4267 raft_consensus.cc:3037] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:16.650817 4267 raft_consensus.cc:504] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:16.650985 4267 leader_election.cc:288] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4 [CANDIDATE]: Term 1 election: Requested vote from peers 5c39d49bf2a04951a2094251619c837d (127.1.226.193:41701), f881bfbdccd44104bddfce14b5d0ba2a (127.1.226.194:43337)
I20260928 04:43:16.651140 3811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bceda5b45c384571a84d046af258c194" candidate_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c39d49bf2a04951a2094251619c837d"
I20260928 04:43:16.651217 3811 raft_consensus.cc:3037] T bceda5b45c384571a84d046af258c194 P 5c39d49bf2a04951a2094251619c837d [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:16.651180 3950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bceda5b45c384571a84d046af258c194" candidate_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f881bfbdccd44104bddfce14b5d0ba2a"
I20260928 04:43:16.651265 3950 raft_consensus.cc:3037] T bceda5b45c384571a84d046af258c194 P f881bfbdccd44104bddfce14b5d0ba2a [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:16.651930 3811 raft_consensus.cc:2454] T bceda5b45c384571a84d046af258c194 P 5c39d49bf2a04951a2094251619c837d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ac7e274cd244f9c8ab79ab71210daf4 in term 1.
I20260928 04:43:16.651949 3950 raft_consensus.cc:2454] T bceda5b45c384571a84d046af258c194 P f881bfbdccd44104bddfce14b5d0ba2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ac7e274cd244f9c8ab79ab71210daf4 in term 1.
I20260928 04:43:16.652117 4024 leader_election.cc:302] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ac7e274cd244f9c8ab79ab71210daf4, 5c39d49bf2a04951a2094251619c837d; no voters:
I20260928 04:43:16.652279 4267 raft_consensus.cc:2781] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:16.652415 4267 raft_consensus.cc:686] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 1 LEADER]: Becoming Leader. State: Replica: 0ac7e274cd244f9c8ab79ab71210daf4, State: Running, Role: LEADER
I20260928 04:43:16.652509 4267 consensus_queue.cc:237] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } }
I20260928 04:43:16.653206 3652 catalog_manager.cc:5495] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ac7e274cd244f9c8ab79ab71210daf4 (127.1.226.195). New cstate: current_term: 1 leader_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ac7e274cd244f9c8ab79ab71210daf4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43749 } health_report { overall_health: HEALTHY } } }
I20260928 04:43:16.662086 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:16.659597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46134 (local address 127.1.226.195:43749)
0928 04:43:16.659820 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:16.659821 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:16.659827 (+ 6us) server_negotiation.cc:407] Connection header received
0928 04:43:16.659844 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:16.659844 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:16.659854 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:16.659930 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:43:16.660064 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:16.660616 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:16.661362 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:16.661576 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:16.661651 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:16.661805 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:43:16.661847 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:43:16.661902 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:16.661971 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 04:43:16.661992 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":152,"threads_started":1}
I20260928 04:43:16.664570 3811 raft_consensus.cc:1261] T bceda5b45c384571a84d046af258c194 P 5c39d49bf2a04951a2094251619c837d [term 1 FOLLOWER]: Refusing update from remote peer 0ac7e274cd244f9c8ab79ab71210daf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:16.664726 3950 raft_consensus.cc:1261] T bceda5b45c384571a84d046af258c194 P f881bfbdccd44104bddfce14b5d0ba2a [term 1 FOLLOWER]: Refusing update from remote peer 0ac7e274cd244f9c8ab79ab71210daf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:16.664817 4267 consensus_queue.cc:1035] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c39d49bf2a04951a2094251619c837d" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:16.664959 4277 consensus_queue.cc:1035] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f881bfbdccd44104bddfce14b5d0ba2a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 04:43:16.669731 4233 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 04:43:18.188606 3652 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:43:18.188704 3652 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:43:18.189631 3652 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6955b3f0c50e4c0cb1864be5c8d8eb41: Sending DeleteTablet for 3 replicas of tablet bceda5b45c384571a84d046af258c194
I20260928 04:43:18.189904 3791 tablet_service.cc:1514] Processing DeleteTablet for tablet bceda5b45c384571a84d046af258c194 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:18 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:56014
I20260928 04:43:18.189913 4071 tablet_service.cc:1514] Processing DeleteTablet for tablet bceda5b45c384571a84d046af258c194 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:18 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:58604
I20260928 04:43:18.189919 3931 tablet_service.cc:1514] Processing DeleteTablet for tablet bceda5b45c384571a84d046af258c194 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:18 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:44952
I20260928 04:43:18.190105 4306 tablet_replica.cc:330] T bceda5b45c384571a84d046af258c194 P f881bfbdccd44104bddfce14b5d0ba2a: stopping tablet replica
I20260928 04:43:18.190158 4307 tablet_replica.cc:330] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4: stopping tablet replica
I20260928 04:43:18.190171 4306 raft_consensus.cc:2229] T bceda5b45c384571a84d046af258c194 P f881bfbdccd44104bddfce14b5d0ba2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:43:18.190212 4307 raft_consensus.cc:2229] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:43:18.190243 4306 raft_consensus.cc:2258] T bceda5b45c384571a84d046af258c194 P f881bfbdccd44104bddfce14b5d0ba2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:18.190295 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 3726
I20260928 04:43:18.191639 4307 raft_consensus.cc:2258] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:18.192414 4306 ts_tablet_manager.cc:1905] T bceda5b45c384571a84d046af258c194 P f881bfbdccd44104bddfce14b5d0ba2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:18.194442 4306 ts_tablet_manager.cc:1918] T bceda5b45c384571a84d046af258c194 P f881bfbdccd44104bddfce14b5d0ba2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 04:43:18.194510 4306 log.cc:1196] T bceda5b45c384571a84d046af258c194 P f881bfbdccd44104bddfce14b5d0ba2a: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-1/wal/wals/bceda5b45c384571a84d046af258c194
I20260928 04:43:18.194729 4306 ts_tablet_manager.cc:1939] T bceda5b45c384571a84d046af258c194 P f881bfbdccd44104bddfce14b5d0ba2a: Deleting consensus metadata
I20260928 04:43:18.195117 3638 catalog_manager.cc:4841] TS f881bfbdccd44104bddfce14b5d0ba2a (127.1.226.194:43337): tablet bceda5b45c384571a84d046af258c194 (table test-table [id=f907683cc4fc43459e3fe0eb7354b5c5]) successfully deleted
I20260928 04:43:18.196337 4307 ts_tablet_manager.cc:1905] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 04:43:18.197007 3640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.226.193:41701 (error 108)
I20260928 04:43:18.197638 4310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:18.197420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:41701 (local address 127.0.0.1:43132)
0928 04:43:18.197560 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:18.197607 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.193:41701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260928 04:43:18.197639 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 3866
I20260928 04:43:18.198360 4307 ts_tablet_manager.cc:1918] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 04:43:18.198411 4307 log.cc:1196] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570585693787-1931-0/minicluster-data/ts-2/wal/wals/bceda5b45c384571a84d046af258c194
I20260928 04:43:18.198625 4307 ts_tablet_manager.cc:1939] T bceda5b45c384571a84d046af258c194 P 0ac7e274cd244f9c8ab79ab71210daf4: Deleting consensus metadata
I20260928 04:43:18.198946 3639 catalog_manager.cc:4841] TS 0ac7e274cd244f9c8ab79ab71210daf4 (127.1.226.195:43749): tablet bceda5b45c384571a84d046af258c194 (table test-table [id=f907683cc4fc43459e3fe0eb7354b5c5]) successfully deleted
W20260928 04:43:18.203341 3640 catalog_manager.cc:4584] TS 5c39d49bf2a04951a2094251619c837d (127.1.226.193:41701): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bceda5b45c384571a84d046af258c194: Network error: Client connection negotiation failed: client connection to 127.1.226.193:41701: connect: Connection refused (error 111)
I20260928 04:43:18.204535 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 4006
I20260928 04:43:18.210413 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 3622
2026-09-28T04:43:18Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6169 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:43:18 dist-test-slave-j5c3 krb5kdc[4313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:43:18 dist-test-slave-j5c3 krb5kdc[4313](info): set up 2 sockets
Sep 28 04:43:18 dist-test-slave-j5c3 krb5kdc[4313](info): commencing operation
krb5kdc: starting...
W20260928 04:43:19.360085 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:43:19 dist-test-slave-j5c3 krb5kdc[4313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:43:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:43:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:43:19.481387 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:33417
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:45461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:33417
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:19.565124 4326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:19.565487 4326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:19.565534 4326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:19.566843 4326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:43:19.566895 4326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:19.566908 4326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:19.566921 4326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:43:19.566934 4326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:43:19.568567 4326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:45461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:33417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:33417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:19.568850 4326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:19.569114 4326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:19.574797 4331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:19.574904 4326 server_base.cc:1034] running on GCE node
W20260928 04:43:19.574791 4334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:19.574887 4332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:19.575357 4326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:19.575682 4326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:19.576874 4326 hybrid_clock.cc:648] HybridClock initialized: now 1790570599576865 us; error 36 us; skew 500 ppm
Sep 28 04:43:19 dist-test-slave-j5c3 krb5kdc[4313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570599, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:19.579540 4326 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:43:19.580044 4326 webserver.cc:460] Webserver started at http://127.1.226.254:33829/ using document root <none> and password file <none>
I20260928 04:43:19.580338 4326 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:19.580451 4326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:19.580615 4326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:19.581724 4326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "17a0c3b0e4e4460591de0d901e91f0b7"
format_stamp: "Formatted at 2026-09-28 04:43:19 on dist-test-slave-j5c3"
server_key: "5b1ab1e364f8bdca501cdc1feef837a4"
server_key_iv: "64971bf3b21bd6bfb1fdff97fa929320"
server_key_version: "encryptionkey@0"
I20260928 04:43:19.582103 4326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "17a0c3b0e4e4460591de0d901e91f0b7"
format_stamp: "Formatted at 2026-09-28 04:43:19 on dist-test-slave-j5c3"
server_key: "5b1ab1e364f8bdca501cdc1feef837a4"
server_key_iv: "64971bf3b21bd6bfb1fdff97fa929320"
server_key_version: "encryptionkey@0"
I20260928 04:43:19.583475 4326 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:43:19.584309 4341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:19.584502 4326 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:19.584605 4326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "17a0c3b0e4e4460591de0d901e91f0b7"
format_stamp: "Formatted at 2026-09-28 04:43:19 on dist-test-slave-j5c3"
server_key: "5b1ab1e364f8bdca501cdc1feef837a4"
server_key_iv: "64971bf3b21bd6bfb1fdff97fa929320"
server_key_version: "encryptionkey@0"
I20260928 04:43:19.584692 4326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:19.601749 4326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:19.604334 4326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:19.604550 4326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:19.608997 4326 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:33417
I20260928 04:43:19.609046 4393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:33417 every 8 connection(s)
I20260928 04:43:19.609501 4326 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:43:19.610472 4394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:19.612843 4394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17a0c3b0e4e4460591de0d901e91f0b7: Bootstrap starting.
I20260928 04:43:19.613389 4394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17a0c3b0e4e4460591de0d901e91f0b7: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:19.613621 4394 log.cc:824] T 00000000000000000000000000000000 P 17a0c3b0e4e4460591de0d901e91f0b7: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:19.614280 4394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17a0c3b0e4e4460591de0d901e91f0b7: No bootstrap required, opened a new log
I20260928 04:43:19.615808 4394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17a0c3b0e4e4460591de0d901e91f0b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a0c3b0e4e4460591de0d901e91f0b7" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33417 } }
I20260928 04:43:19.615947 4394 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17a0c3b0e4e4460591de0d901e91f0b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:19.615968 4394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17a0c3b0e4e4460591de0d901e91f0b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17a0c3b0e4e4460591de0d901e91f0b7, State: Initialized, Role: FOLLOWER
I20260928 04:43:19.616127 4394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17a0c3b0e4e4460591de0d901e91f0b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a0c3b0e4e4460591de0d901e91f0b7" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33417 } }
I20260928 04:43:19.616248 4394 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17a0c3b0e4e4460591de0d901e91f0b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:43:19.616287 4394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17a0c3b0e4e4460591de0d901e91f0b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:43:19.616329 4394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17a0c3b0e4e4460591de0d901e91f0b7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:19.616565 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 4326
I20260928 04:43:19.616653 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:43:19.616842 1931 external_mini_cluster.cc:1428] Setting key 71309bc94ed297e07a36f635c4d21d8e
I20260928 04:43:19.617069 4394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17a0c3b0e4e4460591de0d901e91f0b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a0c3b0e4e4460591de0d901e91f0b7" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33417 } }
I20260928 04:43:19.617178 4394 leader_election.cc:302] T 00000000000000000000000000000000 P 17a0c3b0e4e4460591de0d901e91f0b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17a0c3b0e4e4460591de0d901e91f0b7; no voters:
I20260928 04:43:19.617357 4394 leader_election.cc:288] T 00000000000000000000000000000000 P 17a0c3b0e4e4460591de0d901e91f0b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:43:19.617379 4397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17a0c3b0e4e4460591de0d901e91f0b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:19.617491 4397 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17a0c3b0e4e4460591de0d901e91f0b7 [term 1 LEADER]: Becoming Leader. State: Replica: 17a0c3b0e4e4460591de0d901e91f0b7, State: Running, Role: LEADER
I20260928 04:43:19.617570 4397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17a0c3b0e4e4460591de0d901e91f0b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a0c3b0e4e4460591de0d901e91f0b7" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33417 } }
I20260928 04:43:19.617630 4394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17a0c3b0e4e4460591de0d901e91f0b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:43:19.617904 4400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17a0c3b0e4e4460591de0d901e91f0b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17a0c3b0e4e4460591de0d901e91f0b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a0c3b0e4e4460591de0d901e91f0b7" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33417 } } }
I20260928 04:43:19.617904 4401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17a0c3b0e4e4460591de0d901e91f0b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17a0c3b0e4e4460591de0d901e91f0b7. Latest consensus state: current_term: 1 leader_uuid: "17a0c3b0e4e4460591de0d901e91f0b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a0c3b0e4e4460591de0d901e91f0b7" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33417 } } }
I20260928 04:43:19.618106 4401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17a0c3b0e4e4460591de0d901e91f0b7 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:19.618095 4400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17a0c3b0e4e4460591de0d901e91f0b7 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:19.618909 4403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:43:19.619438 4403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:43:19.621407 4403 catalog_manager.cc:1347] Generated new cluster ID: c8c7da9ed3e1484ea6ad662ab9bbcb6d
I20260928 04:43:19.621462 4403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:43:19 dist-test-slave-j5c3 krb5kdc[4313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:19.628034 4399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:19.617336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33678 (local address 127.1.226.254:33417)
0928 04:43:19.618201 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:19.618211 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:19.618243 (+ 32us) server_negotiation.cc:407] Connection header received
0928 04:43:19.618379 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:19.618387 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:19.618457 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:19.618620 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:19.619482 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:19.620697 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:19.621723 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:19.622155 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:19.622276 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:19.624622 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:19.624626 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:19.624626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:19.624640 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:19.626405 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:19.626803 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:19.626810 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:19.626811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:19.626911 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:19.627129 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:19.627130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:19.627131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:19.627387 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:19.627559 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:19.627657 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 04:43:19.627816 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":815,"thread_start_us":73,"threads_started":1}
I20260928 04:43:19.633862 4403 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:43:19.634389 4403 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:43:19.647667 4403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17a0c3b0e4e4460591de0d901e91f0b7: Generated new TSK 0
I20260928 04:43:19.647918 4403 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.226.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
I20260928 04:43:19.681468 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:0
--local_ip_for_outbound_sockets=127.1.226.193
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:33417
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:45461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:19.766861 4420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:19.767108 4420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:19.767136 4420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:19.768451 4420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:19.768507 4420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:19.768537 4420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:43:19.770185 4420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:45461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:33417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:19.770478 4420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:19.770848 4420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:19.776264 4426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:19.776495 4420 server_base.cc:1034] running on GCE node
W20260928 04:43:19.776264 4428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:19.776288 4425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:19.776957 4420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:19.777292 4420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:19.778475 4420 hybrid_clock.cc:648] HybridClock initialized: now 1790570599778460 us; error 53 us; skew 500 ppm
Sep 28 04:43:19 dist-test-slave-j5c3 krb5kdc[4313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570599, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:19.781339 4420 init.cc:377] Logged in from keytab as kudu/127.1.226.193@KRBTEST.COM (short username kudu)
I20260928 04:43:19.781771 4420 webserver.cc:460] Webserver started at http://127.1.226.193:42507/ using document root <none> and password file <none>
I20260928 04:43:19.782009 4420 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:19.782055 4420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:19.782208 4420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:19.783288 4420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/data/instance:
uuid: "483ccad7dece4d94a8dcfe83ecd59828"
format_stamp: "Formatted at 2026-09-28 04:43:19 on dist-test-slave-j5c3"
server_key: "29a8b51dfe940447930f1ba069bc3c13"
server_key_iv: "4cc32d52d08ac18a4725e816c63830dc"
server_key_version: "encryptionkey@0"
I20260928 04:43:19.783643 4420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance:
uuid: "483ccad7dece4d94a8dcfe83ecd59828"
format_stamp: "Formatted at 2026-09-28 04:43:19 on dist-test-slave-j5c3"
server_key: "29a8b51dfe940447930f1ba069bc3c13"
server_key_iv: "4cc32d52d08ac18a4725e816c63830dc"
server_key_version: "encryptionkey@0"
I20260928 04:43:19.785074 4420 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:43:19.785879 4435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:19.786172 4420 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:19.786311 4420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "483ccad7dece4d94a8dcfe83ecd59828"
format_stamp: "Formatted at 2026-09-28 04:43:19 on dist-test-slave-j5c3"
server_key: "29a8b51dfe940447930f1ba069bc3c13"
server_key_iv: "4cc32d52d08ac18a4725e816c63830dc"
server_key_version: "encryptionkey@0"
I20260928 04:43:19.786393 4420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:19.796262 4420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:19.798738 4420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:19.798906 4420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:19.799139 4420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:19.799474 4420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:19.799525 4420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:19.799567 4420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:19.799592 4420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:19.805500 4420 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:45381
I20260928 04:43:19.805549 4548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:45381 every 8 connection(s)
I20260928 04:43:19.805888 4420 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
I20260928 04:43:19.807132 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 4420
I20260928 04:43:19.807216 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance
I20260928 04:43:19.807404 1931 external_mini_cluster.cc:1428] Setting key 03829f37d4be2e6db925318a43961639
Sep 28 04:43:19 dist-test-slave-j5c3 krb5kdc[4313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570599, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.226.194@KRBTEST.COM; defaulting to no policy
I20260928 04:43:19.816092 4399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:19.806626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:36565 (local address 127.1.226.254:33417)
0928 04:43:19.806766 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:19.806768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:19.806952 (+ 184us) server_negotiation.cc:407] Connection header received
0928 04:43:19.807617 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:19.807619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:19.807635 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:19.807701 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:19.808760 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:19.809361 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:19.810373 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:19.810610 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:19.810701 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:19.813236 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:19.813238 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:19.813238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:19.813245 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:19.814632 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:19.815097 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:19.815098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:19.815098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:19.815153 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:19.815425 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:19.815426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:19.815426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:19.815537 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:19.815611 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:19.815914 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 04:43:19.815999 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 04:43:19.816344 4551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:19.806607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:33417 (local address 127.1.226.193:36565)
0928 04:43:19.806802 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:19.806858 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:19.807501 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:19.807775 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:19.807779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:19.808454 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:19.808658 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:19.808663 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:19.809468 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:19.809469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:19.810289 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:19.810292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:19.810701 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:19.810703 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:19.810839 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:19.811402 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:19.811412 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:19.813100 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:19.814791 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:19.814797 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:19.814798 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:19.815010 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:19.815233 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:19.815234 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:19.815234 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:19.815348 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:19.815602 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:19.815608 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:19.815777 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 04:43:19.815891 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 04:43:19.816052 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":93,"threads_started":1}
I20260928 04:43:19.816843 4549 heartbeater.cc:347] Connected to a master server at 127.1.226.254:33417
I20260928 04:43:19.816964 4549 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:19.817150 4549 heartbeater.cc:511] Master 127.1.226.254:33417 requested a full tablet report, sending...
I20260928 04:43:19.817764 4358 ts_manager.cc:194] Registered new tserver with Master: 483ccad7dece4d94a8dcfe83ecd59828 (127.1.226.193:45381)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.194@KRBTEST.COM" created.
I20260928 04:43:19.818624 4358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.193@KRBTEST.COM'} at 127.1.226.193:36565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
I20260928 04:43:19.832952 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:0
--local_ip_for_outbound_sockets=127.1.226.194
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:33417
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:45461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:19.915467 4554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:19.915673 4554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:19.915696 4554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:19.916927 4554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:19.916975 4554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:19.917054 4554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:43:19.918664 4554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:45461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:33417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:19.918926 4554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:19.919171 4554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:19.924190 4562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:19.924227 4560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:19.924190 4559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:19.924515 4554 server_base.cc:1034] running on GCE node
I20260928 04:43:19.924744 4554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:19.924985 4554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:19.926144 4554 hybrid_clock.cc:648] HybridClock initialized: now 1790570599926141 us; error 39 us; skew 500 ppm
Sep 28 04:43:19 dist-test-slave-j5c3 krb5kdc[4313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570599, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:19.928879 4554 init.cc:377] Logged in from keytab as kudu/127.1.226.194@KRBTEST.COM (short username kudu)
I20260928 04:43:19.929345 4554 webserver.cc:460] Webserver started at http://127.1.226.194:40983/ using document root <none> and password file <none>
I20260928 04:43:19.929580 4554 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:19.929647 4554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:19.929797 4554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:19.930929 4554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/data/instance:
uuid: "6af3485823994823ba01a285477069fc"
format_stamp: "Formatted at 2026-09-28 04:43:19 on dist-test-slave-j5c3"
server_key: "5c5c8f7c7580e59f696d8e3a05173d1a"
server_key_iv: "ad13af5fc0cbb71ceebf772d1847312e"
server_key_version: "encryptionkey@0"
I20260928 04:43:19.931311 4554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance:
uuid: "6af3485823994823ba01a285477069fc"
format_stamp: "Formatted at 2026-09-28 04:43:19 on dist-test-slave-j5c3"
server_key: "5c5c8f7c7580e59f696d8e3a05173d1a"
server_key_iv: "ad13af5fc0cbb71ceebf772d1847312e"
server_key_version: "encryptionkey@0"
I20260928 04:43:19.932886 4554 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:43:19.933743 4569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:19.933935 4554 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:43:19.934039 4554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "6af3485823994823ba01a285477069fc"
format_stamp: "Formatted at 2026-09-28 04:43:19 on dist-test-slave-j5c3"
server_key: "5c5c8f7c7580e59f696d8e3a05173d1a"
server_key_iv: "ad13af5fc0cbb71ceebf772d1847312e"
server_key_version: "encryptionkey@0"
I20260928 04:43:19.934114 4554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:19.955806 4554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:19.959177 4554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:19.959355 4554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:19.959631 4554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:19.959926 4554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:19.959957 4554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:19.959980 4554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:19.960000 4554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:19.966202 4554 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:42237
I20260928 04:43:19.966248 4682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:42237 every 8 connection(s)
I20260928 04:43:19.966625 4554 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
I20260928 04:43:19.968580 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 4554
I20260928 04:43:19.968777 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance
I20260928 04:43:19.968984 1931 external_mini_cluster.cc:1428] Setting key 7676a5565faacfb54347a4102f3d1730
Sep 28 04:43:19 dist-test-slave-j5c3 krb5kdc[4313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570599, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.226.195@KRBTEST.COM; defaulting to no policy
I20260928 04:43:19.977056 4399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:19.967442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:48973 (local address 127.1.226.254:33417)
0928 04:43:19.967514 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:19.967516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:19.967793 (+ 277us) server_negotiation.cc:407] Connection header received
0928 04:43:19.968413 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:19.968414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:19.968427 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:19.968519 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:19.969783 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:19.970550 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:19.971610 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:19.971811 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:19.971896 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:19.974181 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:19.974183 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:19.974183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:19.974195 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:19.975599 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:19.976044 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:19.976046 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:19.976046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:19.976104 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:19.976381 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:19.976382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:19.976382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:19.976501 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:19.976574 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:19.976869 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 04:43:19.976962 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 04:43:19.977267 4685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:19.967431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:33417 (local address 127.1.226.194:48973)
0928 04:43:19.967642 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:19.967691 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:19.968280 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:19.968604 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:19.968608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:19.969439 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:19.969714 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:19.969720 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:19.970645 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:19.970646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:19.971496 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:19.971498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:19.971906 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:19.971908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:19.972048 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:19.972702 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:19.972711 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:19.974065 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:19.975696 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:19.975703 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:19.975706 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:19.975951 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:19.976185 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:19.976186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:19.976186 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:19.976309 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:19.976565 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:19.976572 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:19.976721 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 04:43:19.976833 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 04:43:19.976999 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":99,"threads_started":1}
I20260928 04:43:19.977728 4683 heartbeater.cc:347] Connected to a master server at 127.1.226.254:33417
I20260928 04:43:19.977821 4683 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:19.977978 4683 heartbeater.cc:511] Master 127.1.226.254:33417 requested a full tablet report, sending...
I20260928 04:43:19.978377 4358 ts_manager.cc:194] Registered new tserver with Master: 6af3485823994823ba01a285477069fc (127.1.226.194:42237)
I20260928 04:43:19.978771 4358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.194@KRBTEST.COM'} at 127.1.226.194:48973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
I20260928 04:43:19.995103 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:0
--local_ip_for_outbound_sockets=127.1.226.195
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:33417
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:45461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:20.077507 4688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:20.077775 4688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:20.077813 4688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:20.079000 4688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:20.079074 4688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:20.079118 4688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:43:20.080590 4688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:45461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:33417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:20.080873 4688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:20.081107 4688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:43:20.086319 4688 server_base.cc:1034] running on GCE node
W20260928 04:43:20.086302 4694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:20.086282 4693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:20.086297 4696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:20.086705 4688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:20.086928 4688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:20.088100 4688 hybrid_clock.cc:648] HybridClock initialized: now 1790570600088085 us; error 56 us; skew 500 ppm
Sep 28 04:43:20 dist-test-slave-j5c3 krb5kdc[4313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570600, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:20.090881 4688 init.cc:377] Logged in from keytab as kudu/127.1.226.195@KRBTEST.COM (short username kudu)
I20260928 04:43:20.091357 4688 webserver.cc:460] Webserver started at http://127.1.226.195:34375/ using document root <none> and password file <none>
I20260928 04:43:20.091594 4688 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:20.091660 4688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:20.091812 4688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:20.092995 4688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/data/instance:
uuid: "a69e36fb1ec241fbbd714f19420459cf"
format_stamp: "Formatted at 2026-09-28 04:43:20 on dist-test-slave-j5c3"
server_key: "162af01f21bd6f0acce340fdb73ed122"
server_key_iv: "cea92d5c060e19de6072758b742a8249"
server_key_version: "encryptionkey@0"
I20260928 04:43:20.093365 4688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance:
uuid: "a69e36fb1ec241fbbd714f19420459cf"
format_stamp: "Formatted at 2026-09-28 04:43:20 on dist-test-slave-j5c3"
server_key: "162af01f21bd6f0acce340fdb73ed122"
server_key_iv: "cea92d5c060e19de6072758b742a8249"
server_key_version: "encryptionkey@0"
I20260928 04:43:20.094758 4688 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:43:20.095546 4703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:20.095733 4688 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:43:20.095831 4688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "a69e36fb1ec241fbbd714f19420459cf"
format_stamp: "Formatted at 2026-09-28 04:43:20 on dist-test-slave-j5c3"
server_key: "162af01f21bd6f0acce340fdb73ed122"
server_key_iv: "cea92d5c060e19de6072758b742a8249"
server_key_version: "encryptionkey@0"
I20260928 04:43:20.095917 4688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:20.119331 4688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:20.121865 4688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:20.122040 4688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:20.122299 4688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:20.122622 4688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:20.122686 4688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:20.122728 4688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:20.122756 4688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:20.128931 4688 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:34005
I20260928 04:43:20.129009 4816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:34005 every 8 connection(s)
I20260928 04:43:20.129369 4688 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
I20260928 04:43:20.130280 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 4688
I20260928 04:43:20.130383 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance
I20260928 04:43:20.130577 1931 external_mini_cluster.cc:1428] Setting key 3c00da350b974520e6c96ad79d14fb08
Sep 28 04:43:20 dist-test-slave-j5c3 krb5kdc[4313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570600, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:20.139127 4399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:20.130181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:35221 (local address 127.1.226.254:33417)
0928 04:43:20.130238 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:20.130241 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:20.130436 (+ 195us) server_negotiation.cc:407] Connection header received
0928 04:43:20.131006 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:20.131007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:20.131020 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:20.131090 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:20.132271 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:20.132828 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:20.133841 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:20.134023 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:20.134078 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:20.136276 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:20.136278 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:20.136278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:20.136286 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:20.137770 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:20.138276 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:20.138277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:20.138277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:20.138333 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:20.138553 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:20.138553 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:20.138554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:20.138660 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:20.138709 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:20.138988 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 04:43:20.139060 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 04:43:20.139304 4819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:20.130175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:33417 (local address 127.1.226.195:35221)
0928 04:43:20.130321 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:20.130362 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:20.130898 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:20.131255 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:20.131259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:20.131905 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:20.132104 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:20.132109 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:20.132915 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:20.132916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:20.133739 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:20.133741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:20.134092 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:20.134093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:20.134217 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:20.134679 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:20.134689 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:20.136102 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:20.137893 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:20.137900 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:20.137901 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:20.138209 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:20.138395 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:20.138395 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:20.138396 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:20.138499 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:20.138713 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:20.138716 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:20.138874 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 04:43:20.138960 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 04:43:20.139085 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260928 04:43:20.139806 4817 heartbeater.cc:347] Connected to a master server at 127.1.226.254:33417
I20260928 04:43:20.139936 4817 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:20.140125 4817 heartbeater.cc:511] Master 127.1.226.254:33417 requested a full tablet report, sending...
I20260928 04:43:20.140552 4358 ts_manager.cc:194] Registered new tserver with Master: a69e36fb1ec241fbbd714f19420459cf (127.1.226.195:34005)
I20260928 04:43:20.140930 4358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.195@KRBTEST.COM'} at 127.1.226.195:35221
I20260928 04:43:20.142287 1931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 04:43:20.153774 4399 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:20.152654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33680 (local address 127.1.226.254:33417)
0928 04:43:20.152745 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:20.152748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:20.152876 (+ 128us) server_negotiation.cc:407] Connection header received
0928 04:43:20.152978 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:20.152979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:20.152991 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:20.153061 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:20.153695 (+ 634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33680: BlockingRecv error: recv got EOF from 127.0.0.1:33680 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 04:43:20.154665 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 4420
I20260928 04:43:20.159768 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 4554
I20260928 04:43:20.164285 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 4688
I20260928 04:43:20.168833 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 4326
2026-09-28T04:43:20Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1957 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:43:20 dist-test-slave-j5c3 krb5kdc[4829](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:43:20 dist-test-slave-j5c3 krb5kdc[4829](info): set up 2 sockets
Sep 28 04:43:20 dist-test-slave-j5c3 krb5kdc[4829](info): commencing operation
krb5kdc: starting...
W20260928 04:43:21.236747 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:43:21 dist-test-slave-j5c3 krb5kdc[4829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:43:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:43:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:43:21.362344 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:37961
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:37961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:21.445184 4842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:21.445479 4842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:21.445521 4842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:21.446727 4842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:43:21.446805 4842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:21.446831 4842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:21.446863 4842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:43:21.446888 4842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:43:21.448580 4842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:37961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:37961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:21.448865 4842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:21.449105 4842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:21.454250 4850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:21.454311 4848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:21.454296 4847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:21.455062 4842 server_base.cc:1034] running on GCE node
I20260928 04:43:21.455350 4842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:21.455653 4842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:21.456877 4842 hybrid_clock.cc:648] HybridClock initialized: now 1790570601456851 us; error 40 us; skew 500 ppm
Sep 28 04:43:21 dist-test-slave-j5c3 krb5kdc[4829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570601, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:21.466630 4842 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:43:21.467260 4842 webserver.cc:460] Webserver started at http://127.1.226.254:34747/ using document root <none> and password file <none>
I20260928 04:43:21.467536 4842 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:21.467609 4842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:21.467759 4842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:21.468989 4842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "aaebdb523dbe410ba64d9002b3ea01d5"
format_stamp: "Formatted at 2026-09-28 04:43:21 on dist-test-slave-j5c3"
server_key: "4938301f9798a45062c52ccabcca7568"
server_key_iv: "bedcd284d2ee94739f70aab7043b4d2b"
server_key_version: "encryptionkey@0"
I20260928 04:43:21.469381 4842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "aaebdb523dbe410ba64d9002b3ea01d5"
format_stamp: "Formatted at 2026-09-28 04:43:21 on dist-test-slave-j5c3"
server_key: "4938301f9798a45062c52ccabcca7568"
server_key_iv: "bedcd284d2ee94739f70aab7043b4d2b"
server_key_version: "encryptionkey@0"
I20260928 04:43:21.470935 4842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:43:21.471750 4857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:21.471963 4842 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:21.472074 4842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "aaebdb523dbe410ba64d9002b3ea01d5"
format_stamp: "Formatted at 2026-09-28 04:43:21 on dist-test-slave-j5c3"
server_key: "4938301f9798a45062c52ccabcca7568"
server_key_iv: "bedcd284d2ee94739f70aab7043b4d2b"
server_key_version: "encryptionkey@0"
I20260928 04:43:21.472150 4842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:21.489686 4842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:21.492089 4842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:21.492295 4842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:21.496505 4842 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:37961
I20260928 04:43:21.496541 4909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:37961 every 8 connection(s)
I20260928 04:43:21.496953 4842 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:43:21.497485 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 4842
I20260928 04:43:21.497601 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:43:21.497821 1931 external_mini_cluster.cc:1428] Setting key 63121a35bdb28e7a48ef06e096e05f42
I20260928 04:43:21.498071 4910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:21.501523 4910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaebdb523dbe410ba64d9002b3ea01d5: Bootstrap starting.
I20260928 04:43:21.502314 4910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aaebdb523dbe410ba64d9002b3ea01d5: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:21.502617 4910 log.cc:824] T 00000000000000000000000000000000 P aaebdb523dbe410ba64d9002b3ea01d5: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:21.503317 4910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaebdb523dbe410ba64d9002b3ea01d5: No bootstrap required, opened a new log
Sep 28 04:43:21 dist-test-slave-j5c3 krb5kdc[4829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:21.505694 4910 raft_consensus.cc:348] T 00000000000000000000000000000000 P aaebdb523dbe410ba64d9002b3ea01d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaebdb523dbe410ba64d9002b3ea01d5" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 37961 } }
I20260928 04:43:21.505868 4910 raft_consensus.cc:374] T 00000000000000000000000000000000 P aaebdb523dbe410ba64d9002b3ea01d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:21.505906 4910 raft_consensus.cc:729] T 00000000000000000000000000000000 P aaebdb523dbe410ba64d9002b3ea01d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaebdb523dbe410ba64d9002b3ea01d5, State: Initialized, Role: FOLLOWER
I20260928 04:43:21.506040 4910 consensus_queue.cc:260] T 00000000000000000000000000000000 P aaebdb523dbe410ba64d9002b3ea01d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaebdb523dbe410ba64d9002b3ea01d5" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 37961 } }
I20260928 04:43:21.506132 4910 raft_consensus.cc:388] T 00000000000000000000000000000000 P aaebdb523dbe410ba64d9002b3ea01d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:43:21.506211 4910 raft_consensus.cc:482] T 00000000000000000000000000000000 P aaebdb523dbe410ba64d9002b3ea01d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:43:21.506263 4910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aaebdb523dbe410ba64d9002b3ea01d5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:21.506934 4910 raft_consensus.cc:504] T 00000000000000000000000000000000 P aaebdb523dbe410ba64d9002b3ea01d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaebdb523dbe410ba64d9002b3ea01d5" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 37961 } }
I20260928 04:43:21.507022 4910 leader_election.cc:302] T 00000000000000000000000000000000 P aaebdb523dbe410ba64d9002b3ea01d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aaebdb523dbe410ba64d9002b3ea01d5; no voters:
I20260928 04:43:21.507233 4910 leader_election.cc:288] T 00000000000000000000000000000000 P aaebdb523dbe410ba64d9002b3ea01d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:43:21.507284 4915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aaebdb523dbe410ba64d9002b3ea01d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:21.507463 4915 raft_consensus.cc:686] T 00000000000000000000000000000000 P aaebdb523dbe410ba64d9002b3ea01d5 [term 1 LEADER]: Becoming Leader. State: Replica: aaebdb523dbe410ba64d9002b3ea01d5, State: Running, Role: LEADER
I20260928 04:43:21.507508 4910 sys_catalog.cc:564] T 00000000000000000000000000000000 P aaebdb523dbe410ba64d9002b3ea01d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:43:21.507555 4915 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaebdb523dbe410ba64d9002b3ea01d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaebdb523dbe410ba64d9002b3ea01d5" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 37961 } }
I20260928 04:43:21.507901 4915 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaebdb523dbe410ba64d9002b3ea01d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aaebdb523dbe410ba64d9002b3ea01d5. Latest consensus state: current_term: 1 leader_uuid: "aaebdb523dbe410ba64d9002b3ea01d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaebdb523dbe410ba64d9002b3ea01d5" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 37961 } } }
I20260928 04:43:21.507912 4916 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaebdb523dbe410ba64d9002b3ea01d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aaebdb523dbe410ba64d9002b3ea01d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaebdb523dbe410ba64d9002b3ea01d5" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 37961 } } }
I20260928 04:43:21.507988 4915 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaebdb523dbe410ba64d9002b3ea01d5 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:21.508015 4916 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaebdb523dbe410ba64d9002b3ea01d5 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:21.509438 4913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:21.498398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40872 (local address 127.1.226.254:37961)
0928 04:43:21.498569 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:21.498581 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:21.498612 (+ 31us) server_negotiation.cc:407] Connection header received
0928 04:43:21.499255 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:21.499262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:21.499336 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:21.499459 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:21.500296 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:21.501172 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:21.502225 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:21.502705 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:21.502847 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:21.505589 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:21.505592 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:21.505593 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:21.505604 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:21.507642 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:21.508094 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:21.508097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:21.508098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:21.508247 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:21.508485 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:21.508486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:21.508487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:21.508728 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:21.508887 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:21.509001 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 04:43:21.509138 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
I20260928 04:43:21.509200 4922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:43:21.509950 4922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:43:21.510326 1931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:43:21.511412 4922 catalog_manager.cc:1347] Generated new cluster ID: 602983de68674ad6bf99b48aa630e1ee
I20260928 04:43:21.511463 4922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:43:21.527706 4913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:21.524069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40886 (local address 127.1.226.254:37961)
0928 04:43:21.524141 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:21.524143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:21.524325 (+ 182us) server_negotiation.cc:407] Connection header received
0928 04:43:21.524393 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:21.524394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:21.524406 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:21.524481 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:21.524875 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:21.525419 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:21.526159 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:21.526386 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:21.526469 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:21.527235 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:21.527236 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:21.527237 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:43:21.527439 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:21.527456 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:43:21.527462 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:21.527530 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:21.527591 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 04:43:21.527615 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 04:43:21.553236 4922 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:43:21.554020 4922 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:43:21.562170 4922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aaebdb523dbe410ba64d9002b3ea01d5: Generated new TSK 0
I20260928 04:43:21.562403 4922 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:43:21.568753 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 4842
2026-09-28T04:43:21Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1397 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:43:21 dist-test-slave-j5c3 krb5kdc[4943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:43:21 dist-test-slave-j5c3 krb5kdc[4943](info): set up 2 sockets
Sep 28 04:43:21 dist-test-slave-j5c3 krb5kdc[4943](info): commencing operation
krb5kdc: starting...
W20260928 04:43:22.603979 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:43:22 dist-test-slave-j5c3 krb5kdc[4943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:43:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:43:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:43:22.723356 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:41421
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:41421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:22.805496 4956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:22.805763 4956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:22.805799 4956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:22.806962 4956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:43:22.807025 4956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:22.807049 4956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:22.807070 4956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:43:22.807101 4956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:43:22.808640 4956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:41421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:41421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:22.808926 4956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:22.809170 4956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:43:22.814051 4956 server_base.cc:1034] running on GCE node
W20260928 04:43:22.814016 4962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:22.814015 4964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:22.814030 4961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:22.814565 4956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:22.814824 4956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:22.816015 4956 hybrid_clock.cc:648] HybridClock initialized: now 1790570602816004 us; error 28 us; skew 500 ppm
Sep 28 04:43:22 dist-test-slave-j5c3 krb5kdc[4943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570602, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:22.818455 4956 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:43:22.818953 4956 webserver.cc:460] Webserver started at http://127.1.226.254:33819/ using document root <none> and password file <none>
I20260928 04:43:22.819196 4956 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:22.819263 4956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:22.819418 4956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:22.820561 4956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "522489a4d8b9471c9f9370f8e7875f9f"
format_stamp: "Formatted at 2026-09-28 04:43:22 on dist-test-slave-j5c3"
server_key: "70adcea8d9d58ce7f6cc7aeb848ad0f6"
server_key_iv: "31c484faea631075f3442dd03d51c578"
server_key_version: "encryptionkey@0"
I20260928 04:43:22.820925 4956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "522489a4d8b9471c9f9370f8e7875f9f"
format_stamp: "Formatted at 2026-09-28 04:43:22 on dist-test-slave-j5c3"
server_key: "70adcea8d9d58ce7f6cc7aeb848ad0f6"
server_key_iv: "31c484faea631075f3442dd03d51c578"
server_key_version: "encryptionkey@0"
I20260928 04:43:22.822238 4956 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:43:22.822964 4971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:22.823134 4956 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:43:22.823240 4956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "522489a4d8b9471c9f9370f8e7875f9f"
format_stamp: "Formatted at 2026-09-28 04:43:22 on dist-test-slave-j5c3"
server_key: "70adcea8d9d58ce7f6cc7aeb848ad0f6"
server_key_iv: "31c484faea631075f3442dd03d51c578"
server_key_version: "encryptionkey@0"
I20260928 04:43:22.823319 4956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:22.834110 4956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:22.836270 4956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:22.836436 4956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:22.840783 4956 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:41421
I20260928 04:43:22.840826 5023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:41421 every 8 connection(s)
I20260928 04:43:22.841229 4956 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:43:22.842248 5024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:22.844756 5024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 522489a4d8b9471c9f9370f8e7875f9f: Bootstrap starting.
I20260928 04:43:22.845384 5024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 522489a4d8b9471c9f9370f8e7875f9f: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:22.845677 5024 log.cc:824] T 00000000000000000000000000000000 P 522489a4d8b9471c9f9370f8e7875f9f: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:22.846467 5024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 522489a4d8b9471c9f9370f8e7875f9f: No bootstrap required, opened a new log
I20260928 04:43:22.848230 5024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 522489a4d8b9471c9f9370f8e7875f9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522489a4d8b9471c9f9370f8e7875f9f" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 41421 } }
I20260928 04:43:22.848559 5024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 522489a4d8b9471c9f9370f8e7875f9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:22.848567 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 4956
I20260928 04:43:22.848613 5024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 522489a4d8b9471c9f9370f8e7875f9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522489a4d8b9471c9f9370f8e7875f9f, State: Initialized, Role: FOLLOWER
I20260928 04:43:22.848657 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:43:22.848791 5024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 522489a4d8b9471c9f9370f8e7875f9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522489a4d8b9471c9f9370f8e7875f9f" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 41421 } }
I20260928 04:43:22.848893 5024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 522489a4d8b9471c9f9370f8e7875f9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:43:22.848928 5024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 522489a4d8b9471c9f9370f8e7875f9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:43:22.848935 1931 external_mini_cluster.cc:1428] Setting key 5a87e482f3ffa6cddce650c1aea0fadc
I20260928 04:43:22.848985 5024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 522489a4d8b9471c9f9370f8e7875f9f [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:22.849884 5024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 522489a4d8b9471c9f9370f8e7875f9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522489a4d8b9471c9f9370f8e7875f9f" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 41421 } }
I20260928 04:43:22.850020 5024 leader_election.cc:302] T 00000000000000000000000000000000 P 522489a4d8b9471c9f9370f8e7875f9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 522489a4d8b9471c9f9370f8e7875f9f; no voters:
I20260928 04:43:22.850236 5024 leader_election.cc:288] T 00000000000000000000000000000000 P 522489a4d8b9471c9f9370f8e7875f9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:43:22.850277 5029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 522489a4d8b9471c9f9370f8e7875f9f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:22.850483 5029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 522489a4d8b9471c9f9370f8e7875f9f [term 1 LEADER]: Becoming Leader. State: Replica: 522489a4d8b9471c9f9370f8e7875f9f, State: Running, Role: LEADER
I20260928 04:43:22.850550 5024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 522489a4d8b9471c9f9370f8e7875f9f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:43:22.850610 5029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 522489a4d8b9471c9f9370f8e7875f9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522489a4d8b9471c9f9370f8e7875f9f" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 41421 } }
I20260928 04:43:22.851027 5030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 522489a4d8b9471c9f9370f8e7875f9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "522489a4d8b9471c9f9370f8e7875f9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522489a4d8b9471c9f9370f8e7875f9f" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 41421 } } }
I20260928 04:43:22.851127 5030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 522489a4d8b9471c9f9370f8e7875f9f [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:22.851382 5031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 522489a4d8b9471c9f9370f8e7875f9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 522489a4d8b9471c9f9370f8e7875f9f. Latest consensus state: current_term: 1 leader_uuid: "522489a4d8b9471c9f9370f8e7875f9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522489a4d8b9471c9f9370f8e7875f9f" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 41421 } } }
I20260928 04:43:22.851502 5031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 522489a4d8b9471c9f9370f8e7875f9f [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:22.851740 5037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:43:22.852365 5037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:43:22.854105 5037 catalog_manager.cc:1347] Generated new cluster ID: 8b457f57aa5347769d1f6eaa1df7d549
I20260928 04:43:22.854161 5037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:43:22 dist-test-slave-j5c3 krb5kdc[4943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:22.862556 5028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:22.849481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34540 (local address 127.1.226.254:41421)
0928 04:43:22.849993 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:22.850005 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:22.850044 (+ 39us) server_negotiation.cc:407] Connection header received
0928 04:43:22.850187 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:22.850193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:22.850265 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:22.850428 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:22.851401 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:22.852944 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:22.853997 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:22.854635 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:22.854754 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:22.857723 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:22.857728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:22.857728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:22.857743 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:22.860693 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:22.861154 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:22.861159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:22.861159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:22.861312 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:22.861520 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:22.861521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:22.861521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:22.861907 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:22.862079 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:22.862154 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 04:43:22.862305 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":428,"threads_started":1}
I20260928 04:43:22.872550 5037 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:43:22.873155 5037 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:43:22.882404 5037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 522489a4d8b9471c9f9370f8e7875f9f: Generated new TSK 0
I20260928 04:43:22.882629 5037 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.226.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
I20260928 04:43:22.911391 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:0
--local_ip_for_outbound_sockets=127.1.226.193
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:41421
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:23.000070 5050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:23.000319 5050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:23.000345 5050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:23.001523 5050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:23.001573 5050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:23.001601 5050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:43:23.003126 5050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:41421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:23.003373 5050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:23.003582 5050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:23.009202 5055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:23.009191 5056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:23.009191 5058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:23.009393 5050 server_base.cc:1034] running on GCE node
I20260928 04:43:23.009691 5050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:23.009989 5050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:23.011173 5050 hybrid_clock.cc:648] HybridClock initialized: now 1790570603011164 us; error 39 us; skew 500 ppm
Sep 28 04:43:23 dist-test-slave-j5c3 krb5kdc[4943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570603, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:23.014040 5050 init.cc:377] Logged in from keytab as kudu/127.1.226.193@KRBTEST.COM (short username kudu)
I20260928 04:43:23.014587 5050 webserver.cc:460] Webserver started at http://127.1.226.193:38381/ using document root <none> and password file <none>
I20260928 04:43:23.014858 5050 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:23.014931 5050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:23.015084 5050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:23.016324 5050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/data/instance:
uuid: "cebce9a864c4465c8ea7569e90808acc"
format_stamp: "Formatted at 2026-09-28 04:43:23 on dist-test-slave-j5c3"
server_key: "37a7bb89a9598756bb9d10b6a64fbbf2"
server_key_iv: "920455ce6b0fe7669246fcd67d2f1514"
server_key_version: "encryptionkey@0"
I20260928 04:43:23.016717 5050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance:
uuid: "cebce9a864c4465c8ea7569e90808acc"
format_stamp: "Formatted at 2026-09-28 04:43:23 on dist-test-slave-j5c3"
server_key: "37a7bb89a9598756bb9d10b6a64fbbf2"
server_key_iv: "920455ce6b0fe7669246fcd67d2f1514"
server_key_version: "encryptionkey@0"
I20260928 04:43:23.018069 5050 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:43:23.018796 5065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:23.018977 5050 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:43:23.019078 5050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "cebce9a864c4465c8ea7569e90808acc"
format_stamp: "Formatted at 2026-09-28 04:43:23 on dist-test-slave-j5c3"
server_key: "37a7bb89a9598756bb9d10b6a64fbbf2"
server_key_iv: "920455ce6b0fe7669246fcd67d2f1514"
server_key_version: "encryptionkey@0"
I20260928 04:43:23.019166 5050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:23.048007 5050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:23.050261 5050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:23.050458 5050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:23.050784 5050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:23.051185 5050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:23.051251 5050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:23.051301 5050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:23.051332 5050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:23.057966 5050 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:33945
I20260928 04:43:23.058008 5178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:33945 every 8 connection(s)
I20260928 04:43:23.058372 5050 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
Sep 28 04:43:23 dist-test-slave-j5c3 krb5kdc[4943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570603, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:23.067227 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 5050
I20260928 04:43:23.067355 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance
I20260928 04:43:23.067631 1931 external_mini_cluster.cc:1428] Setting key 1d8d91a38373ad7c91b73a9c8c6591d8
I20260928 04:43:23.068315 5028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:23.059197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:34693 (local address 127.1.226.254:41421)
0928 04:43:23.059268 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:23.059270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:23.059480 (+ 210us) server_negotiation.cc:407] Connection header received
0928 04:43:23.059983 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:23.059984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:23.059994 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:23.060051 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:23.061121 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:23.061636 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:23.062722 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:23.062927 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:23.062981 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:23.065155 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:23.065157 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:23.065157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:23.065164 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:23.066606 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:23.067090 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:23.067091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:23.067091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:23.067141 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:23.067466 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:23.067467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:23.067467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:23.067622 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:23.067694 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:23.068061 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 04:43:23.068200 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 04:43:23.068542 5181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:23.059158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:41421 (local address 127.1.226.193:34693)
0928 04:43:23.059335 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:23.059380 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:23.059891 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:23.060122 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:23.060126 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:23.060808 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:23.061016 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:23.061022 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:23.061740 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:23.061742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:23.062632 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:23.062635 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:23.063008 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:23.063009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:23.063130 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:23.063638 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:23.063646 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:23.065049 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:23.066695 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:23.066702 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:23.066704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:23.067007 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:23.067212 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:23.067213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:23.067213 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:23.067387 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:23.067711 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:23.067720 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:23.067902 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 04:43:23.068040 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 04:43:23.068252 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
I20260928 04:43:23.069180 5179 heartbeater.cc:347] Connected to a master server at 127.1.226.254:41421
I20260928 04:43:23.069307 5179 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:23.069506 5179 heartbeater.cc:511] Master 127.1.226.254:41421 requested a full tablet report, sending...
I20260928 04:43:23.070034 4988 ts_manager.cc:194] Registered new tserver with Master: cebce9a864c4465c8ea7569e90808acc (127.1.226.193:33945)
I20260928 04:43:23.070642 4988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.193@KRBTEST.COM'} at 127.1.226.193:34693
WARNING: no policy specified for kudu/127.1.226.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
I20260928 04:43:23.095196 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:0
--local_ip_for_outbound_sockets=127.1.226.194
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:41421
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:23.178416 5184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:23.178661 5184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:23.178689 5184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:23.179968 5184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:23.180020 5184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:23.180049 5184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:43:23.181639 5184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:41421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:23.181914 5184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:23.182163 5184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:23.187677 5192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:23.187709 5189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:23.188017 5190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:23.188431 5184 server_base.cc:1034] running on GCE node
I20260928 04:43:23.188719 5184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:23.189029 5184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:23.190223 5184 hybrid_clock.cc:648] HybridClock initialized: now 1790570603190201 us; error 35 us; skew 500 ppm
Sep 28 04:43:23 dist-test-slave-j5c3 krb5kdc[4943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570603, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:23.195058 5184 init.cc:377] Logged in from keytab as kudu/127.1.226.194@KRBTEST.COM (short username kudu)
I20260928 04:43:23.195604 5184 webserver.cc:460] Webserver started at http://127.1.226.194:37029/ using document root <none> and password file <none>
I20260928 04:43:23.195899 5184 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:23.195981 5184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:23.196131 5184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:23.197301 5184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/data/instance:
uuid: "d68c2476b0b44db2b1a19b0f1a17afa5"
format_stamp: "Formatted at 2026-09-28 04:43:23 on dist-test-slave-j5c3"
server_key: "0bf069c2e2017f9e8c835d4d32d90925"
server_key_iv: "e7e34bed3b3effcc74c876250e688b95"
server_key_version: "encryptionkey@0"
I20260928 04:43:23.197691 5184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance:
uuid: "d68c2476b0b44db2b1a19b0f1a17afa5"
format_stamp: "Formatted at 2026-09-28 04:43:23 on dist-test-slave-j5c3"
server_key: "0bf069c2e2017f9e8c835d4d32d90925"
server_key_iv: "e7e34bed3b3effcc74c876250e688b95"
server_key_version: "encryptionkey@0"
I20260928 04:43:23.199101 5184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:43:23.200011 5199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:23.200218 5184 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:23.200349 5184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "d68c2476b0b44db2b1a19b0f1a17afa5"
format_stamp: "Formatted at 2026-09-28 04:43:23 on dist-test-slave-j5c3"
server_key: "0bf069c2e2017f9e8c835d4d32d90925"
server_key_iv: "e7e34bed3b3effcc74c876250e688b95"
server_key_version: "encryptionkey@0"
I20260928 04:43:23.200441 5184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:23.213339 5184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:23.215823 5184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:23.216001 5184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:23.216341 5184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:23.216696 5184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:23.216749 5184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:23.216794 5184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:23.216822 5184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:23.223810 5184 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:41239
I20260928 04:43:23.223898 5312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:41239 every 8 connection(s)
I20260928 04:43:23.224310 5184 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
I20260928 04:43:23.230512 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 5184
I20260928 04:43:23.230633 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance
I20260928 04:43:23.230870 1931 external_mini_cluster.cc:1428] Setting key 21da43e8c82b55b4a6a9776718f3230f
Sep 28 04:43:23 dist-test-slave-j5c3 krb5kdc[4943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570603, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:23.236788 5028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:23.225444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:38363 (local address 127.1.226.254:41421)
0928 04:43:23.225512 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:23.225514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:23.225983 (+ 469us) server_negotiation.cc:407] Connection header received
0928 04:43:23.226773 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:23.226774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:23.226787 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:23.226848 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:23.228234 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:23.229082 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:23.230307 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:23.230607 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:23.230684 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:23.233416 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:23.233417 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:23.233418 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:23.233427 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:23.235126 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:23.235673 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:23.235675 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:23.235675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:23.235743 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:23.236011 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:23.236012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:23.236012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:23.236199 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:23.236271 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:23.236544 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 04:43:23.236644 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 04:43:23.236860 5315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:23.225117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:41421 (local address 127.1.226.194:38363)
0928 04:43:23.225822 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:23.225875 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:23.226608 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:23.226935 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:23.226939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:23.227792 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:23.228088 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:23.228095 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:23.229204 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:23.229205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:23.230195 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:23.230197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:23.230762 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:23.230764 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:23.230920 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:23.231529 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:23.231543 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:23.233295 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:23.235246 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:23.235254 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:23.235256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:23.235575 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:23.235821 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:23.235821 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:23.235822 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:23.235938 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:23.236262 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:23.236268 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:23.236399 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 04:43:23.236497 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 04:43:23.236650 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":76,"threads_started":1}
I20260928 04:43:23.237388 5313 heartbeater.cc:347] Connected to a master server at 127.1.226.254:41421
I20260928 04:43:23.237497 5313 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:23.237718 5313 heartbeater.cc:511] Master 127.1.226.254:41421 requested a full tablet report, sending...
I20260928 04:43:23.238153 4988 ts_manager.cc:194] Registered new tserver with Master: d68c2476b0b44db2b1a19b0f1a17afa5 (127.1.226.194:41239)
I20260928 04:43:23.238536 4988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.194@KRBTEST.COM'} at 127.1.226.194:38363
WARNING: no policy specified for kudu/127.1.226.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
I20260928 04:43:23.258718 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:0
--local_ip_for_outbound_sockets=127.1.226.195
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:41421
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:23.342782 5318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:23.342998 5318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:23.343024 5318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:23.344298 5318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:23.344342 5318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:23.344370 5318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:43:23.345813 5318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:41421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:23.346007 5318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:23.346194 5318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:23.351367 5324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:23.351567 5326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:23.351581 5318 server_base.cc:1034] running on GCE node
W20260928 04:43:23.351594 5323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:23.351948 5318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:23.352238 5318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:23.353410 5318 hybrid_clock.cc:648] HybridClock initialized: now 1790570603353401 us; error 41 us; skew 500 ppm
Sep 28 04:43:23 dist-test-slave-j5c3 krb5kdc[4943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570603, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:23.356053 5318 init.cc:377] Logged in from keytab as kudu/127.1.226.195@KRBTEST.COM (short username kudu)
I20260928 04:43:23.356519 5318 webserver.cc:460] Webserver started at http://127.1.226.195:39711/ using document root <none> and password file <none>
I20260928 04:43:23.356765 5318 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:23.356812 5318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:23.356976 5318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:23.358089 5318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/data/instance:
uuid: "3c52c68c60c2415aa414ac15801b29bf"
format_stamp: "Formatted at 2026-09-28 04:43:23 on dist-test-slave-j5c3"
server_key: "011c9b1169f612685258afccfa00406e"
server_key_iv: "edfe3421d2a8e522a2845002ee3267b9"
server_key_version: "encryptionkey@0"
I20260928 04:43:23.358440 5318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance:
uuid: "3c52c68c60c2415aa414ac15801b29bf"
format_stamp: "Formatted at 2026-09-28 04:43:23 on dist-test-slave-j5c3"
server_key: "011c9b1169f612685258afccfa00406e"
server_key_iv: "edfe3421d2a8e522a2845002ee3267b9"
server_key_version: "encryptionkey@0"
I20260928 04:43:23.359814 5318 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:43:23.360564 5333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:23.360714 5318 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:23.360823 5318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "3c52c68c60c2415aa414ac15801b29bf"
format_stamp: "Formatted at 2026-09-28 04:43:23 on dist-test-slave-j5c3"
server_key: "011c9b1169f612685258afccfa00406e"
server_key_iv: "edfe3421d2a8e522a2845002ee3267b9"
server_key_version: "encryptionkey@0"
I20260928 04:43:23.360881 5318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:23.378841 5318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:23.381197 5318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:23.381321 5318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:23.381512 5318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:23.381841 5318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:23.381875 5318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:23.381896 5318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:23.381943 5318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:23.387882 5318 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:37261
I20260928 04:43:23.387980 5446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:37261 every 8 connection(s)
I20260928 04:43:23.388325 5318 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
I20260928 04:43:23.393939 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 5318
I20260928 04:43:23.394048 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance
I20260928 04:43:23.394224 1931 external_mini_cluster.cc:1428] Setting key 2b36b13b43dc3842787285e6d02a6a44
Sep 28 04:43:23 dist-test-slave-j5c3 krb5kdc[4943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570603, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:23.398335 5028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:23.389279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:36553 (local address 127.1.226.254:41421)
0928 04:43:23.389366 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:23.389368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:23.389598 (+ 230us) server_negotiation.cc:407] Connection header received
0928 04:43:23.390117 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:23.390118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:23.390127 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:23.390184 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:23.391194 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:23.391680 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:23.392865 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:23.393062 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:23.393113 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:23.395647 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:23.395648 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:23.395649 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:23.395656 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:23.397087 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:23.397523 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:23.397523 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:23.397524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:23.397573 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:23.397799 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:23.397799 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:23.397800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:23.397901 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:23.397951 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:23.398193 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 04:43:23.398254 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 04:43:23.398584 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:23.389258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:41421 (local address 127.1.226.195:36553)
0928 04:43:23.389452 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:23.389499 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:23.390031 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:23.390260 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:23.390264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:23.390899 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:23.391109 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:23.391115 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:23.391761 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:23.391762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:23.392772 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:23.392774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:23.393166 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:23.393167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:23.393300 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:23.393902 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:23.393911 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:23.395557 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:23.397193 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:23.397200 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:23.397202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:23.397455 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:23.397631 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:23.397632 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:23.397632 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:23.397742 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:23.397955 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:23.397959 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:23.398082 (+ 123us) client_negotiation.cc:769] Sending connection context
0928 04:43:23.398169 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 04:43:23.398333 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":113,"threads_started":1}
I20260928 04:43:23.399123 5447 heartbeater.cc:347] Connected to a master server at 127.1.226.254:41421
I20260928 04:43:23.399245 5447 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:23.399407 5447 heartbeater.cc:511] Master 127.1.226.254:41421 requested a full tablet report, sending...
I20260928 04:43:23.399776 4988 ts_manager.cc:194] Registered new tserver with Master: 3c52c68c60c2415aa414ac15801b29bf (127.1.226.195:37261)
I20260928 04:43:23.400200 4988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.195@KRBTEST.COM'} at 127.1.226.195:36553
I20260928 04:43:23.406279 1931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 04:43:23 dist-test-slave-j5c3 krb5kdc[4943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570603, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 04:43:23 dist-test-slave-j5c3 krb5kdc[4943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570603, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:23.431515 5028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:23.423813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34546 (local address 127.1.226.254:41421)
0928 04:43:23.423877 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:23.423879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:23.424042 (+ 163us) server_negotiation.cc:407] Connection header received
0928 04:43:23.424178 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:23.424179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:23.424192 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:23.424265 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:23.424861 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:23.425363 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:23.426225 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:23.426427 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:23.426517 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:23.429092 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:23.429093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:23.429093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:23.429101 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:23.430383 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:23.430751 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:23.430751 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:23.430752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:23.430808 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:23.430990 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:23.430990 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:23.430991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:23.431138 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:23.431217 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:23.431344 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 04:43:23.431408 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 04:43:23.431842 4988 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34546
I20260928 04:43:23.432452 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 5050
I20260928 04:43:23.437479 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 5184
I20260928 04:43:23.442555 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 5318
I20260928 04:43:23.447057 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 4956
2026-09-28T04:43:23Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1881 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:43:23 dist-test-slave-j5c3 krb5kdc[5459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:43:23 dist-test-slave-j5c3 krb5kdc[5459](info): set up 2 sockets
Sep 28 04:43:23 dist-test-slave-j5c3 krb5kdc[5459](info): commencing operation
krb5kdc: starting...
W20260928 04:43:24.492312 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:43:24 dist-test-slave-j5c3 krb5kdc[5459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:43:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:43:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:43:24.615074 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:39355
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:40573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:39355
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:24.698849 5472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:24.699070 5472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:24.699093 5472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:24.700356 5472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:43:24.700403 5472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:24.700418 5472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:24.700431 5472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:43:24.700451 5472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:43:24.702061 5472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:40573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:39355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:39355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:24.702335 5472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:24.702590 5472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:24.707981 5477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:24.707973 5480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:24.707993 5478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:24.708211 5472 server_base.cc:1034] running on GCE node
I20260928 04:43:24.708498 5472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:24.708804 5472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:24.709957 5472 hybrid_clock.cc:648] HybridClock initialized: now 1790570604709924 us; error 54 us; skew 500 ppm
Sep 28 04:43:24 dist-test-slave-j5c3 krb5kdc[5459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570604, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:24.712677 5472 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:43:24.713147 5472 webserver.cc:460] Webserver started at http://127.1.226.254:40133/ using document root <none> and password file <none>
I20260928 04:43:24.713346 5472 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:24.713387 5472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:24.713475 5472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:24.714514 5472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "d9eeaaf562b8422c90eae99ef7bbb6d3"
format_stamp: "Formatted at 2026-09-28 04:43:24 on dist-test-slave-j5c3"
server_key: "e015836bd5b04ba874968246f0dd5c7a"
server_key_iv: "1ab0d1fdf68458910864e4f927fcb395"
server_key_version: "encryptionkey@0"
I20260928 04:43:24.714834 5472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "d9eeaaf562b8422c90eae99ef7bbb6d3"
format_stamp: "Formatted at 2026-09-28 04:43:24 on dist-test-slave-j5c3"
server_key: "e015836bd5b04ba874968246f0dd5c7a"
server_key_iv: "1ab0d1fdf68458910864e4f927fcb395"
server_key_version: "encryptionkey@0"
I20260928 04:43:24.716144 5472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 04:43:24.716923 5487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:24.717095 5472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:43:24.717201 5472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "d9eeaaf562b8422c90eae99ef7bbb6d3"
format_stamp: "Formatted at 2026-09-28 04:43:24 on dist-test-slave-j5c3"
server_key: "e015836bd5b04ba874968246f0dd5c7a"
server_key_iv: "1ab0d1fdf68458910864e4f927fcb395"
server_key_version: "encryptionkey@0"
I20260928 04:43:24.717276 5472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:24.733243 5472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:24.735351 5472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:24.735524 5472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:24.739928 5472 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:39355
I20260928 04:43:24.740034 5539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:39355 every 8 connection(s)
I20260928 04:43:24.740550 5472 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:43:24.741285 5540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:24.743633 5540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9eeaaf562b8422c90eae99ef7bbb6d3: Bootstrap starting.
I20260928 04:43:24.744297 5540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9eeaaf562b8422c90eae99ef7bbb6d3: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:24.744550 5540 log.cc:824] T 00000000000000000000000000000000 P d9eeaaf562b8422c90eae99ef7bbb6d3: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:24.745239 5540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9eeaaf562b8422c90eae99ef7bbb6d3: No bootstrap required, opened a new log
I20260928 04:43:24.746814 5540 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9eeaaf562b8422c90eae99ef7bbb6d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9eeaaf562b8422c90eae99ef7bbb6d3" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 39355 } }
I20260928 04:43:24.746955 5540 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9eeaaf562b8422c90eae99ef7bbb6d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:24.746979 5540 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9eeaaf562b8422c90eae99ef7bbb6d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9eeaaf562b8422c90eae99ef7bbb6d3, State: Initialized, Role: FOLLOWER
I20260928 04:43:24.747083 5540 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9eeaaf562b8422c90eae99ef7bbb6d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9eeaaf562b8422c90eae99ef7bbb6d3" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 39355 } }
I20260928 04:43:24.747135 5540 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9eeaaf562b8422c90eae99ef7bbb6d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:43:24.747155 5540 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9eeaaf562b8422c90eae99ef7bbb6d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:43:24.747187 5540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9eeaaf562b8422c90eae99ef7bbb6d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:24.748070 5540 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9eeaaf562b8422c90eae99ef7bbb6d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9eeaaf562b8422c90eae99ef7bbb6d3" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 39355 } }
I20260928 04:43:24.748212 5540 leader_election.cc:302] T 00000000000000000000000000000000 P d9eeaaf562b8422c90eae99ef7bbb6d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9eeaaf562b8422c90eae99ef7bbb6d3; no voters:
I20260928 04:43:24.748401 5540 leader_election.cc:288] T 00000000000000000000000000000000 P d9eeaaf562b8422c90eae99ef7bbb6d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:43:24.748431 5543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9eeaaf562b8422c90eae99ef7bbb6d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:24.748549 5543 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9eeaaf562b8422c90eae99ef7bbb6d3 [term 1 LEADER]: Becoming Leader. State: Replica: d9eeaaf562b8422c90eae99ef7bbb6d3, State: Running, Role: LEADER
I20260928 04:43:24.748631 5543 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9eeaaf562b8422c90eae99ef7bbb6d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9eeaaf562b8422c90eae99ef7bbb6d3" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 39355 } }
I20260928 04:43:24.748667 5540 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9eeaaf562b8422c90eae99ef7bbb6d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:43:24.748998 5545 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9eeaaf562b8422c90eae99ef7bbb6d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9eeaaf562b8422c90eae99ef7bbb6d3. Latest consensus state: current_term: 1 leader_uuid: "d9eeaaf562b8422c90eae99ef7bbb6d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9eeaaf562b8422c90eae99ef7bbb6d3" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 39355 } } }
I20260928 04:43:24.749028 5544 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9eeaaf562b8422c90eae99ef7bbb6d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9eeaaf562b8422c90eae99ef7bbb6d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9eeaaf562b8422c90eae99ef7bbb6d3" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 39355 } } }
I20260928 04:43:24.749218 5544 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9eeaaf562b8422c90eae99ef7bbb6d3 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:24.749171 5545 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9eeaaf562b8422c90eae99ef7bbb6d3 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:24.749612 5550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:43:24.750173 5550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:43:24.750232 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 5472
I20260928 04:43:24.750347 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:43:24.750533 1931 external_mini_cluster.cc:1428] Setting key ca3fa941ff9a61825ebca86cdaf77650
I20260928 04:43:24.752503 5550 catalog_manager.cc:1347] Generated new cluster ID: ff393ac3747e42c9aa24528cba56dd8d
I20260928 04:43:24.752554 5550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:43:24 dist-test-slave-j5c3 krb5kdc[5459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:24.762918 5562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:24.751228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46792 (local address 127.1.226.254:39355)
0928 04:43:24.751599 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:24.751608 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:24.751632 (+ 24us) server_negotiation.cc:407] Connection header received
0928 04:43:24.751729 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:24.751735 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:24.751778 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:24.751938 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:24.752763 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:24.754249 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:24.755302 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:24.755982 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:24.756108 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:24.758997 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:24.759003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:24.759003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:24.759018 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:24.761078 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:24.761531 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:24.761534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:24.761535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:24.761659 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:24.761876 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:24.761877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:24.761877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:24.762201 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:24.762375 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:24.762502 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 04:43:24.762646 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":86,"threads_started":1}
I20260928 04:43:24.765161 5550 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:43:24.765714 5550 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:43:24.770885 5550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9eeaaf562b8422c90eae99ef7bbb6d3: Generated new TSK 0
I20260928 04:43:24.771111 5550 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.226.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
I20260928 04:43:24.800019 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:0
--local_ip_for_outbound_sockets=127.1.226.193
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:39355
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:40573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:24.884521 5566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:24.884755 5566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:24.884783 5566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:24.885994 5566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:24.886047 5566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:24.886077 5566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:43:24.887674 5566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:40573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:39355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:24.887938 5566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:24.888242 5566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:24.893396 5572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:24.893391 5571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:24.893391 5574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:24.893739 5566 server_base.cc:1034] running on GCE node
I20260928 04:43:24.893941 5566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:24.894153 5566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:24.895299 5566 hybrid_clock.cc:648] HybridClock initialized: now 1790570604895287 us; error 30 us; skew 500 ppm
Sep 28 04:43:24 dist-test-slave-j5c3 krb5kdc[5459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570604, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:24.898083 5566 init.cc:377] Logged in from keytab as kudu/127.1.226.193@KRBTEST.COM (short username kudu)
I20260928 04:43:24.898556 5566 webserver.cc:460] Webserver started at http://127.1.226.193:44369/ using document root <none> and password file <none>
I20260928 04:43:24.898857 5566 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:24.898932 5566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:24.899080 5566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:24.900224 5566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-0/data/instance:
uuid: "d111166a9f8e41bf8845329f6dbd6d8b"
format_stamp: "Formatted at 2026-09-28 04:43:24 on dist-test-slave-j5c3"
server_key: "bc2822c48fca7f97eaa19780cc7e6be8"
server_key_iv: "070995ecc395a063fec2e7732474fa08"
server_key_version: "encryptionkey@0"
I20260928 04:43:24.900614 5566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance:
uuid: "d111166a9f8e41bf8845329f6dbd6d8b"
format_stamp: "Formatted at 2026-09-28 04:43:24 on dist-test-slave-j5c3"
server_key: "bc2822c48fca7f97eaa19780cc7e6be8"
server_key_iv: "070995ecc395a063fec2e7732474fa08"
server_key_version: "encryptionkey@0"
I20260928 04:43:24.901996 5566 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:43:24.902810 5581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:24.903008 5566 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:43:24.903121 5566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "d111166a9f8e41bf8845329f6dbd6d8b"
format_stamp: "Formatted at 2026-09-28 04:43:24 on dist-test-slave-j5c3"
server_key: "bc2822c48fca7f97eaa19780cc7e6be8"
server_key_iv: "070995ecc395a063fec2e7732474fa08"
server_key_version: "encryptionkey@0"
I20260928 04:43:24.903206 5566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:24.947806 5566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:24.950383 5566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:24.950564 5566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:24.950824 5566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:24.951151 5566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:24.951207 5566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:24.951252 5566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:24.951282 5566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:24.957525 5566 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:37557
I20260928 04:43:24.957587 5694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:37557 every 8 connection(s)
I20260928 04:43:24.958020 5566 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
Sep 28 04:43:24 dist-test-slave-j5c3 krb5kdc[5459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570604, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:24.966012 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 5566
I20260928 04:43:24.966152 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance
I20260928 04:43:24.966337 1931 external_mini_cluster.cc:1428] Setting key 960208eea5e055bdc08bbdaae65441c2
I20260928 04:43:24.968549 5562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:24.958913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:43099 (local address 127.1.226.254:39355)
0928 04:43:24.958977 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:24.958979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:24.959140 (+ 161us) server_negotiation.cc:407] Connection header received
0928 04:43:24.960141 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:24.960143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:24.960176 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:24.960236 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:24.960906 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:24.961410 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:24.962470 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:24.962716 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:24.962776 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:24.965144 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:24.965145 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:24.965146 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:24.965155 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:24.966783 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:24.967224 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:24.967226 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:24.967226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:24.967295 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:24.967603 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:24.967604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:24.967604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:24.967784 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:24.967868 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:24.968292 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 04:43:24.968399 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 04:43:24.968716 5697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:24.958830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:39355 (local address 127.1.226.193:43099)
0928 04:43:24.958998 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:24.959043 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:24.960021 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:24.960326 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:24.960330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:24.960600 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:24.960810 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:24.960820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:24.961496 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:24.961497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:24.962382 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:24.962385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:24.962813 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:24.962814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:24.962940 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:24.963477 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:24.963485 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:24.965045 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:24.966842 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:24.966850 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:24.966853 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:24.967138 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:24.967374 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:24.967375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:24.967375 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:24.967523 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:24.967881 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:24.967887 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:24.968042 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 04:43:24.968247 (+ 205us) client_negotiation.cc:240] Negotiation successful
0928 04:43:24.968433 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
I20260928 04:43:24.969476 5695 heartbeater.cc:347] Connected to a master server at 127.1.226.254:39355
I20260928 04:43:24.969578 5695 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:24.969794 5695 heartbeater.cc:511] Master 127.1.226.254:39355 requested a full tablet report, sending...
I20260928 04:43:24.970336 5504 ts_manager.cc:194] Registered new tserver with Master: d111166a9f8e41bf8845329f6dbd6d8b (127.1.226.193:37557)
I20260928 04:43:24.971036 5504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.193@KRBTEST.COM'} at 127.1.226.193:43099
WARNING: no policy specified for kudu/127.1.226.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
I20260928 04:43:24.995890 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:0
--local_ip_for_outbound_sockets=127.1.226.194
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:39355
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:40573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:25.083441 5700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:25.083659 5700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:25.083684 5700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:25.084831 5700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:25.084873 5700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:25.084899 5700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:43:25.086338 5700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:40573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:39355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:25.086524 5700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:25.086699 5700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:25.091692 5705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:25.091810 5706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:25.091692 5708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:25.091981 5700 server_base.cc:1034] running on GCE node
I20260928 04:43:25.092312 5700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:25.092567 5700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:25.093740 5700 hybrid_clock.cc:648] HybridClock initialized: now 1790570605093734 us; error 32 us; skew 500 ppm
Sep 28 04:43:25 dist-test-slave-j5c3 krb5kdc[5459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570605, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:25.096421 5700 init.cc:377] Logged in from keytab as kudu/127.1.226.194@KRBTEST.COM (short username kudu)
I20260928 04:43:25.096904 5700 webserver.cc:460] Webserver started at http://127.1.226.194:44671/ using document root <none> and password file <none>
I20260928 04:43:25.097157 5700 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:25.097234 5700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:25.097384 5700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:25.098462 5700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-1/data/instance:
uuid: "137aade113b04d9aaec44f1d897aecb7"
format_stamp: "Formatted at 2026-09-28 04:43:25 on dist-test-slave-j5c3"
server_key: "57b8a7c235d8aa1eba742ad96b63a661"
server_key_iv: "faba13ff90917b9f857322e426fb57c7"
server_key_version: "encryptionkey@0"
I20260928 04:43:25.098832 5700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance:
uuid: "137aade113b04d9aaec44f1d897aecb7"
format_stamp: "Formatted at 2026-09-28 04:43:25 on dist-test-slave-j5c3"
server_key: "57b8a7c235d8aa1eba742ad96b63a661"
server_key_iv: "faba13ff90917b9f857322e426fb57c7"
server_key_version: "encryptionkey@0"
I20260928 04:43:25.100327 5700 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:43:25.101106 5715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:25.101305 5700 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:43:25.101405 5700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "137aade113b04d9aaec44f1d897aecb7"
format_stamp: "Formatted at 2026-09-28 04:43:25 on dist-test-slave-j5c3"
server_key: "57b8a7c235d8aa1eba742ad96b63a661"
server_key_iv: "faba13ff90917b9f857322e426fb57c7"
server_key_version: "encryptionkey@0"
I20260928 04:43:25.101492 5700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:25.117093 5700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:25.119136 5700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:25.119314 5700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:25.119622 5700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:25.119992 5700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:25.120047 5700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:25.120087 5700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:25.120115 5700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:25.126123 5700 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:45391
I20260928 04:43:25.126183 5828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:45391 every 8 connection(s)
I20260928 04:43:25.126545 5700 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
I20260928 04:43:25.131950 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 5700
I20260928 04:43:25.132085 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance
I20260928 04:43:25.132309 1931 external_mini_cluster.cc:1428] Setting key 7d928de81ff28034905e00f341498c4b
Sep 28 04:43:25 dist-test-slave-j5c3 krb5kdc[5459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570605, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:25.137151 5562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:25.127402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:41355 (local address 127.1.226.254:39355)
0928 04:43:25.127468 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:25.127470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:25.127629 (+ 159us) server_negotiation.cc:407] Connection header received
0928 04:43:25.128639 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:25.128640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:25.128650 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:25.128715 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:25.129352 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:25.129862 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:25.130931 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:25.131119 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:25.131205 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:25.133862 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:25.133863 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:25.133863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:25.133870 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:25.135331 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:25.135983 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:25.135984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:25.135985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:25.136051 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:25.136379 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:25.136380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:25.136380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:25.136510 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:25.136591 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:25.136933 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 04:43:25.137025 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 04:43:25.137327 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:25.127320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:39355 (local address 127.1.226.194:41355)
0928 04:43:25.127500 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:25.127543 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:25.128545 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:25.128771 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:25.128774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:25.129031 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:25.129230 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:25.129233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:25.129951 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:25.129952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:25.130838 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:25.130840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:25.131211 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:25.131212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:25.131315 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:25.131885 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:25.131897 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:25.133754 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:25.135431 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:25.135438 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:25.135441 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:25.135852 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:25.136124 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:25.136125 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:25.136125 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:25.136284 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:25.136580 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:25.136587 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:25.136788 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 04:43:25.136896 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 04:43:25.137066 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260928 04:43:25.137775 5829 heartbeater.cc:347] Connected to a master server at 127.1.226.254:39355
I20260928 04:43:25.137900 5829 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:25.138082 5829 heartbeater.cc:511] Master 127.1.226.254:39355 requested a full tablet report, sending...
I20260928 04:43:25.138480 5504 ts_manager.cc:194] Registered new tserver with Master: 137aade113b04d9aaec44f1d897aecb7 (127.1.226.194:45391)
I20260928 04:43:25.138887 5504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.194@KRBTEST.COM'} at 127.1.226.194:41355
WARNING: no policy specified for kudu/127.1.226.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
I20260928 04:43:25.159260 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:0
--local_ip_for_outbound_sockets=127.1.226.195
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:39355
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:40573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:25.248144 5834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:25.248423 5834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:25.248447 5834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:25.249946 5834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:25.250016 5834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:25.250066 5834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:43:25.251736 5834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:40573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:39355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:25.252004 5834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:25.252300 5834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:43:25.257669 5834 server_base.cc:1034] running on GCE node
W20260928 04:43:25.257717 5840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:25.257648 5839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:25.257622 5842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:25.258061 5834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:25.258409 5834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:25.259568 5834 hybrid_clock.cc:648] HybridClock initialized: now 1790570605259517 us; error 69 us; skew 500 ppm
Sep 28 04:43:25 dist-test-slave-j5c3 krb5kdc[5459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570605, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:25.262233 5834 init.cc:377] Logged in from keytab as kudu/127.1.226.195@KRBTEST.COM (short username kudu)
I20260928 04:43:25.263038 5834 webserver.cc:460] Webserver started at http://127.1.226.195:37607/ using document root <none> and password file <none>
I20260928 04:43:25.263329 5834 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:25.263412 5834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:25.263582 5834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:25.264952 5834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-2/data/instance:
uuid: "7d88613a73624a049e9490a866e5e260"
format_stamp: "Formatted at 2026-09-28 04:43:25 on dist-test-slave-j5c3"
server_key: "7e9920aeca8f8ea75f5768b09f0e36b7"
server_key_iv: "3a58a6ee51af288ca979481f2573b432"
server_key_version: "encryptionkey@0"
I20260928 04:43:25.265375 5834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance:
uuid: "7d88613a73624a049e9490a866e5e260"
format_stamp: "Formatted at 2026-09-28 04:43:25 on dist-test-slave-j5c3"
server_key: "7e9920aeca8f8ea75f5768b09f0e36b7"
server_key_iv: "3a58a6ee51af288ca979481f2573b432"
server_key_version: "encryptionkey@0"
I20260928 04:43:25.266909 5834 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:43:25.267789 5849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:25.267997 5834 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:25.268108 5834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "7d88613a73624a049e9490a866e5e260"
format_stamp: "Formatted at 2026-09-28 04:43:25 on dist-test-slave-j5c3"
server_key: "7e9920aeca8f8ea75f5768b09f0e36b7"
server_key_iv: "3a58a6ee51af288ca979481f2573b432"
server_key_version: "encryptionkey@0"
I20260928 04:43:25.268230 5834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:25.276571 5834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:25.278774 5834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:25.278963 5834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:25.279233 5834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:25.279580 5834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:25.279639 5834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:25.279683 5834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:25.279731 5834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:25.286443 5834 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:46449
I20260928 04:43:25.286491 5962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:46449 every 8 connection(s)
I20260928 04:43:25.286856 5834 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
Sep 28 04:43:25 dist-test-slave-j5c3 krb5kdc[5459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570605, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:25.294792 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 5834
I20260928 04:43:25.294910 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance
I20260928 04:43:25.295110 1931 external_mini_cluster.cc:1428] Setting key 54b30a84e0a5a48d757d429ab5241c9d
I20260928 04:43:25.297201 5562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:25.287592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:39593 (local address 127.1.226.254:39355)
0928 04:43:25.287692 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:25.287695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:25.287869 (+ 174us) server_negotiation.cc:407] Connection header received
0928 04:43:25.289020 (+ 1151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:25.289022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:25.289038 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:25.289094 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:25.289820 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:25.290358 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:25.291417 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:25.291627 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:25.291728 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:25.294066 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:25.294072 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:25.294073 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:25.294080 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:25.295757 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:25.296295 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:25.296296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:25.296296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:25.296351 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:25.296576 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:25.296577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:25.296577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:25.296738 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:25.296795 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:25.297050 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 04:43:25.297132 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 04:43:25.297430 5965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:25.287547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:39355 (local address 127.1.226.195:39593)
0928 04:43:25.287704 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:25.287757 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:25.288884 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:25.289192 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:25.289203 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:25.289495 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:25.289703 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:25.289708 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:25.290474 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:25.290475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:25.291310 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:25.291312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:25.291752 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:25.291753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:25.291885 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:25.292464 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:25.292473 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:25.293970 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:25.295842 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:25.295849 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:25.295852 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:25.296220 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:25.296412 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:25.296412 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:25.296413 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:25.296517 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:25.296797 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:25.296801 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:25.296924 (+ 123us) client_negotiation.cc:769] Sending connection context
0928 04:43:25.297005 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 04:43:25.297152 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260928 04:43:25.297941 5963 heartbeater.cc:347] Connected to a master server at 127.1.226.254:39355
I20260928 04:43:25.298043 5963 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:25.298202 5963 heartbeater.cc:511] Master 127.1.226.254:39355 requested a full tablet report, sending...
I20260928 04:43:25.298539 5504 ts_manager.cc:194] Registered new tserver with Master: 7d88613a73624a049e9490a866e5e260 (127.1.226.195:46449)
I20260928 04:43:25.298941 5504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.195@KRBTEST.COM'} at 127.1.226.195:39593
I20260928 04:43:25.306964 1931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 04:43:25 dist-test-slave-j5c3 krb5kdc[5459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 04:43:25 dist-test-slave-j5c3 krb5kdc[5459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.193@KRBTEST.COM
I20260928 04:43:25.330331 5970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:25.322508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49108 (local address 127.1.226.193:37557)
0928 04:43:25.322696 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:25.322701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:25.322714 (+ 13us) server_negotiation.cc:407] Connection header received
0928 04:43:25.322740 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:25.322742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:25.322757 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:25.322830 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:25.323387 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:25.324237 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:25.324963 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:25.325312 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:25.325372 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:25.327660 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:25.327668 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:25.327668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:25.327675 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:25.329314 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:25.329656 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:25.329659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:25.329659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:25.329715 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:25.329886 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:25.329887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:25.329887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:25.330027 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:25.330150 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:25.330189 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 04:43:25.330250 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":89,"threads_started":1}
Sep 28 04:43:25 dist-test-slave-j5c3 krb5kdc[5459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:25.338455 5562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:25.331739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46448 (local address 127.1.226.254:39355)
0928 04:43:25.331852 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:25.331854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:25.331863 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:43:25.331894 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:25.331895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:25.331904 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:25.331977 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:25.332505 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:25.332990 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:25.333636 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:25.333845 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:25.333899 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:25.335905 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:25.335906 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:25.335906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:25.335913 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:25.337420 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:25.337752 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:25.337753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:25.337754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:25.337804 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:25.337972 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:25.337973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:25.337973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:25.338110 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:25.338184 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:25.338271 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 04:43:25.338336 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 04:43:25.338737 5504 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46448
I20260928 04:43:25.339120 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 5566
I20260928 04:43:25.343937 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 5700
I20260928 04:43:25.348065 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 5834
I20260928 04:43:25.352460 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 5472
2026-09-28T04:43:25Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1904 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:43:25 dist-test-slave-j5c3 krb5kdc[5975](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:43:25 dist-test-slave-j5c3 krb5kdc[5975](info): set up 2 sockets
Sep 28 04:43:25 dist-test-slave-j5c3 krb5kdc[5975](info): commencing operation
krb5kdc: starting...
W20260928 04:43:26.400367 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:43:26 dist-test-slave-j5c3 krb5kdc[5975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:43:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:43:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:43:26.543834 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:44805
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:45455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:44805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:26.625833 5988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:26.626061 5988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:26.626085 5988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:26.627300 5988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:43:26.627351 5988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:26.627367 5988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:26.627378 5988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:43:26.627390 5988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:43:26.628995 5988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:45455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:44805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:44805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:26.629262 5988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:26.629524 5988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:43:26.634852 5988 server_base.cc:1034] running on GCE node
W20260928 04:43:26.634825 5996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:26.634811 5993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:26.634828 5994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:26.635205 5988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:26.635468 5988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:26.636636 5988 hybrid_clock.cc:648] HybridClock initialized: now 1790570606636613 us; error 39 us; skew 500 ppm
Sep 28 04:43:26 dist-test-slave-j5c3 krb5kdc[5975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570606, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:26.646571 5988 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:43:26.647047 5988 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:26.647126 5988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:26.647279 5988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:26.648494 5988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "b3a10584a84643b69bb1db59ff1a609f"
format_stamp: "Formatted at 2026-09-28 04:43:26 on dist-test-slave-j5c3"
server_key: "6802f732458cb0a169eaf757a8d7a1a9"
server_key_iv: "612fc0712fa63c25ccb785090d815b0f"
server_key_version: "encryptionkey@0"
I20260928 04:43:26.648869 5988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "b3a10584a84643b69bb1db59ff1a609f"
format_stamp: "Formatted at 2026-09-28 04:43:26 on dist-test-slave-j5c3"
server_key: "6802f732458cb0a169eaf757a8d7a1a9"
server_key_iv: "612fc0712fa63c25ccb785090d815b0f"
server_key_version: "encryptionkey@0"
I20260928 04:43:26.650211 5988 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:43:26.650985 6002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:26.651196 5988 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:26.651302 5988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "b3a10584a84643b69bb1db59ff1a609f"
format_stamp: "Formatted at 2026-09-28 04:43:26 on dist-test-slave-j5c3"
server_key: "6802f732458cb0a169eaf757a8d7a1a9"
server_key_iv: "612fc0712fa63c25ccb785090d815b0f"
server_key_version: "encryptionkey@0"
I20260928 04:43:26.651376 5988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:26.658985 5988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:26.661425 5988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:26.661547 5988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:26.665434 5988 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:44805
I20260928 04:43:26.665457 6054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:44805 every 8 connection(s)
I20260928 04:43:26.665764 5988 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:43:26.666729 6055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:26.669080 6055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3a10584a84643b69bb1db59ff1a609f: Bootstrap starting.
I20260928 04:43:26.669446 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 5988
I20260928 04:43:26.669576 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:43:26.669685 6055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3a10584a84643b69bb1db59ff1a609f: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:26.669808 1931 external_mini_cluster.cc:1428] Setting key 4228dd186fa69a8b43c0dd7d82fd8b83
I20260928 04:43:26.670027 6055 log.cc:824] T 00000000000000000000000000000000 P b3a10584a84643b69bb1db59ff1a609f: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:26.670727 6055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3a10584a84643b69bb1db59ff1a609f: No bootstrap required, opened a new log
I20260928 04:43:26.673107 6055 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3a10584a84643b69bb1db59ff1a609f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a10584a84643b69bb1db59ff1a609f" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44805 } }
I20260928 04:43:26.673280 6055 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3a10584a84643b69bb1db59ff1a609f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:26.673314 6055 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3a10584a84643b69bb1db59ff1a609f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3a10584a84643b69bb1db59ff1a609f, State: Initialized, Role: FOLLOWER
I20260928 04:43:26.673444 6055 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3a10584a84643b69bb1db59ff1a609f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a10584a84643b69bb1db59ff1a609f" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44805 } }
I20260928 04:43:26.673517 6055 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3a10584a84643b69bb1db59ff1a609f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:43:26.673545 6055 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3a10584a84643b69bb1db59ff1a609f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:43:26.673588 6055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3a10584a84643b69bb1db59ff1a609f [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:26.674237 6055 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3a10584a84643b69bb1db59ff1a609f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a10584a84643b69bb1db59ff1a609f" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44805 } }
I20260928 04:43:26.674324 6055 leader_election.cc:302] T 00000000000000000000000000000000 P b3a10584a84643b69bb1db59ff1a609f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3a10584a84643b69bb1db59ff1a609f; no voters:
I20260928 04:43:26.674525 6055 leader_election.cc:288] T 00000000000000000000000000000000 P b3a10584a84643b69bb1db59ff1a609f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:43:26.674578 6060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3a10584a84643b69bb1db59ff1a609f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:26.674743 6060 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3a10584a84643b69bb1db59ff1a609f [term 1 LEADER]: Becoming Leader. State: Replica: b3a10584a84643b69bb1db59ff1a609f, State: Running, Role: LEADER
I20260928 04:43:26.674782 6055 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3a10584a84643b69bb1db59ff1a609f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:43:26.674837 6060 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3a10584a84643b69bb1db59ff1a609f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a10584a84643b69bb1db59ff1a609f" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44805 } }
I20260928 04:43:26.675302 6061 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3a10584a84643b69bb1db59ff1a609f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3a10584a84643b69bb1db59ff1a609f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a10584a84643b69bb1db59ff1a609f" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44805 } } }
I20260928 04:43:26.675407 6061 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3a10584a84643b69bb1db59ff1a609f [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:26.675601 6069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:43:26.675295 6062 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3a10584a84643b69bb1db59ff1a609f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3a10584a84643b69bb1db59ff1a609f. Latest consensus state: current_term: 1 leader_uuid: "b3a10584a84643b69bb1db59ff1a609f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a10584a84643b69bb1db59ff1a609f" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44805 } } }
I20260928 04:43:26.675757 6062 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3a10584a84643b69bb1db59ff1a609f [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:26.676105 6069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 04:43:26 dist-test-slave-j5c3 krb5kdc[5975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:26.677858 6069 catalog_manager.cc:1347] Generated new cluster ID: c135a74568c0457899c6f14f363ab7b1
I20260928 04:43:26.677908 6069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:43:26.681630 6059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:26.670333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54772 (local address 127.1.226.254:44805)
0928 04:43:26.670510 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:26.670520 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:26.670555 (+ 35us) server_negotiation.cc:407] Connection header received
0928 04:43:26.671071 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:26.671079 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:26.671128 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:26.671244 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:26.672009 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:26.672980 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:26.674037 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:26.674573 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:26.674772 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:26.677939 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:26.677943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:26.677943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:26.677955 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:26.679972 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:26.680394 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:26.680397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:26.680398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:26.680506 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:26.680710 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:26.680711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:26.680711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:26.681023 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:26.681197 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:26.681274 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 04:43:26.681409 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
I20260928 04:43:26.687132 6069 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:43:26.687719 6069 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:43:26.691578 6069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3a10584a84643b69bb1db59ff1a609f: Generated new TSK 0
I20260928 04:43:26.691745 6069 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.226.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
I20260928 04:43:26.723330 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:0
--local_ip_for_outbound_sockets=127.1.226.193
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:44805
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:45455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:26.813134 6081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:26.813371 6081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:26.813398 6081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:26.814661 6081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:26.814720 6081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:26.814750 6081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:43:26.816393 6081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:45455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:44805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:26.816675 6081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:26.816948 6081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:26.822182 6087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:26.822312 6081 server_base.cc:1034] running on GCE node
W20260928 04:43:26.822217 6086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:26.822177 6089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:26.822695 6081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:26.822979 6081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:26.824138 6081 hybrid_clock.cc:648] HybridClock initialized: now 1790570606824118 us; error 42 us; skew 500 ppm
Sep 28 04:43:26 dist-test-slave-j5c3 krb5kdc[5975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570606, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:26.826807 6081 init.cc:377] Logged in from keytab as kudu/127.1.226.193@KRBTEST.COM (short username kudu)
I20260928 04:43:26.827157 6081 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:26.827206 6081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:26.827378 6081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:26.828545 6081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-0/data/instance:
uuid: "5ce63c1c94854c668a0698b273f429bc"
format_stamp: "Formatted at 2026-09-28 04:43:26 on dist-test-slave-j5c3"
server_key: "e01e6f24850bd38690487f4d1e014b88"
server_key_iv: "cccfba65e9d8feaf9e46b8ebf245fc06"
server_key_version: "encryptionkey@0"
I20260928 04:43:26.828925 6081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance:
uuid: "5ce63c1c94854c668a0698b273f429bc"
format_stamp: "Formatted at 2026-09-28 04:43:26 on dist-test-slave-j5c3"
server_key: "e01e6f24850bd38690487f4d1e014b88"
server_key_iv: "cccfba65e9d8feaf9e46b8ebf245fc06"
server_key_version: "encryptionkey@0"
I20260928 04:43:26.830267 6081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:26.831082 6095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:26.831274 6081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:26.831348 6081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "5ce63c1c94854c668a0698b273f429bc"
format_stamp: "Formatted at 2026-09-28 04:43:26 on dist-test-slave-j5c3"
server_key: "e01e6f24850bd38690487f4d1e014b88"
server_key_iv: "cccfba65e9d8feaf9e46b8ebf245fc06"
server_key_version: "encryptionkey@0"
I20260928 04:43:26.831403 6081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:26.841472 6081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:26.843469 6081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:26.843639 6081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:26.843948 6081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:26.844350 6081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:26.844389 6081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:26.844413 6081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:26.844426 6081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:26.850253 6081 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:33445
I20260928 04:43:26.850338 6208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:33445 every 8 connection(s)
I20260928 04:43:26.850623 6081 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
Sep 28 04:43:26 dist-test-slave-j5c3 krb5kdc[5975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570606, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:26.859030 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 6081
I20260928 04:43:26.859205 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance
I20260928 04:43:26.859453 1931 external_mini_cluster.cc:1428] Setting key ca34450eaf21f9acba625567342b61a2
I20260928 04:43:26.861065 6059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:26.851556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:39039 (local address 127.1.226.254:44805)
0928 04:43:26.851617 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:26.851619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:26.851889 (+ 270us) server_negotiation.cc:407] Connection header received
0928 04:43:26.852504 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:26.852505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:26.852517 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:26.852575 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:26.853541 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:26.854106 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:26.855155 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:26.855379 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:26.855463 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:26.857828 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:26.857829 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:26.857830 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:26.857839 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:26.859254 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:26.859884 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:26.859886 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:26.859886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:26.859975 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:26.860324 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:26.860325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:26.860326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:26.860446 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:26.860503 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:26.860914 (+ 411us) server_negotiation.cc:299] Negotiation successful
0928 04:43:26.860998 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 04:43:26.861354 6211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:26.851529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44805 (local address 127.1.226.193:39039)
0928 04:43:26.851758 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:26.851798 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:26.852396 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:26.852650 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:26.852654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:26.852918 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:26.853426 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:26.853431 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:26.854213 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:26.854214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:26.855043 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:26.855045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:26.855472 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:26.855473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:26.855603 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:26.856115 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:26.856123 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:26.857663 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:26.859366 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:26.859377 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:26.859380 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:26.859755 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:26.860055 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:26.860056 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:26.860056 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:26.860243 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:26.860519 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:26.860526 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:26.860720 (+ 194us) client_negotiation.cc:769] Sending connection context
0928 04:43:26.860848 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 04:43:26.861051 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":106,"threads_started":1}
I20260928 04:43:26.862141 6209 heartbeater.cc:347] Connected to a master server at 127.1.226.254:44805
I20260928 04:43:26.862243 6209 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:26.862454 6209 heartbeater.cc:511] Master 127.1.226.254:44805 requested a full tablet report, sending...
I20260928 04:43:26.863057 6019 ts_manager.cc:194] Registered new tserver with Master: 5ce63c1c94854c668a0698b273f429bc (127.1.226.193:33445)
I20260928 04:43:26.864013 6019 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.193@KRBTEST.COM'} at 127.1.226.193:39039
WARNING: no policy specified for kudu/127.1.226.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
I20260928 04:43:26.887864 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:0
--local_ip_for_outbound_sockets=127.1.226.194
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:44805
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:45455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:26.969911 6214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:26.970134 6214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:26.970158 6214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:26.971290 6214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:26.971325 6214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:26.971351 6214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:43:26.972911 6214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:45455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:44805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:26.973098 6214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:26.973289 6214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:26.978374 6219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:26.978493 6214 server_base.cc:1034] running on GCE node
W20260928 04:43:26.978351 6222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:26.978581 6220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:26.978813 6214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:26.979100 6214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:26.980239 6214 hybrid_clock.cc:648] HybridClock initialized: now 1790570606980209 us; error 43 us; skew 500 ppm
Sep 28 04:43:26 dist-test-slave-j5c3 krb5kdc[5975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570606, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:26.982838 6214 init.cc:377] Logged in from keytab as kudu/127.1.226.194@KRBTEST.COM (short username kudu)
I20260928 04:43:26.983238 6214 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:26.983309 6214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:26.983448 6214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:26.984589 6214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-1/data/instance:
uuid: "42972670eb0d422db52d9e0af67c9fa9"
format_stamp: "Formatted at 2026-09-28 04:43:26 on dist-test-slave-j5c3"
server_key: "acf29dd96478bc15eb4a07917ea4680e"
server_key_iv: "2f679248bf2681781181b9f535fc0a40"
server_key_version: "encryptionkey@0"
I20260928 04:43:26.984958 6214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance:
uuid: "42972670eb0d422db52d9e0af67c9fa9"
format_stamp: "Formatted at 2026-09-28 04:43:26 on dist-test-slave-j5c3"
server_key: "acf29dd96478bc15eb4a07917ea4680e"
server_key_iv: "2f679248bf2681781181b9f535fc0a40"
server_key_version: "encryptionkey@0"
I20260928 04:43:26.987287 6214 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 04:43:26.988323 6228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:26.988703 6214 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:26.988807 6214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "42972670eb0d422db52d9e0af67c9fa9"
format_stamp: "Formatted at 2026-09-28 04:43:26 on dist-test-slave-j5c3"
server_key: "acf29dd96478bc15eb4a07917ea4680e"
server_key_iv: "2f679248bf2681781181b9f535fc0a40"
server_key_version: "encryptionkey@0"
I20260928 04:43:26.988909 6214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:27.026849 6214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:27.029290 6214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:27.029485 6214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:27.029748 6214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:27.030086 6214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:27.030139 6214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:27.030181 6214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:27.030223 6214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:27.036082 6214 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:41327
I20260928 04:43:27.036118 6341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:41327 every 8 connection(s)
I20260928 04:43:27.036433 6214 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
Sep 28 04:43:27 dist-test-slave-j5c3 krb5kdc[5975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570606, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:27.043484 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 6214
I20260928 04:43:27.043586 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance
I20260928 04:43:27.043826 1931 external_mini_cluster.cc:1428] Setting key 86d8b7f34e52963fc1602dbb548e4224
I20260928 04:43:27.046566 6059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.037195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:38243 (local address 127.1.226.254:44805)
0928 04:43:27.037269 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:27.037271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:27.037504 (+ 233us) server_negotiation.cc:407] Connection header received
0928 04:43:27.038064 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:27.038065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:27.038078 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:27.038123 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:27.039073 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.039571 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.040658 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.040852 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.040927 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:27.043082 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:27.043084 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:27.043085 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:27.043094 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:27.044977 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:27.045413 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:27.045414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:27.045415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:27.045473 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:27.045743 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:27.045744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:27.045744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:27.045914 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:27.045995 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:27.046345 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 04:43:27.046442 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 04:43:27.046661 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.037186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44805 (local address 127.1.226.194:38243)
0928 04:43:27.037377 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:27.037415 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:27.037949 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:27.038203 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:27.038206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:27.038480 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:27.038958 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:27.038963 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.039673 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.039674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:27.040560 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:27.040562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.040932 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.040933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:27.041059 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:27.041592 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:27.041600 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:27.042970 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:27.045064 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:27.045074 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:27.045076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:27.045333 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:27.045548 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:27.045549 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:27.045549 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:27.045674 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:27.046013 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:27.046022 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:27.046195 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 04:43:27.046299 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 04:43:27.046439 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":99,"threads_started":1}
I20260928 04:43:27.047273 6342 heartbeater.cc:347] Connected to a master server at 127.1.226.254:44805
I20260928 04:43:27.047410 6342 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:27.047605 6342 heartbeater.cc:511] Master 127.1.226.254:44805 requested a full tablet report, sending...
I20260928 04:43:27.048023 6019 ts_manager.cc:194] Registered new tserver with Master: 42972670eb0d422db52d9e0af67c9fa9 (127.1.226.194:41327)
I20260928 04:43:27.048604 6019 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.194@KRBTEST.COM'} at 127.1.226.194:38243
WARNING: no policy specified for kudu/127.1.226.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
I20260928 04:43:27.072548 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:0
--local_ip_for_outbound_sockets=127.1.226.195
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:44805
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:45455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:27.152578 6347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:27.152786 6347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:27.152810 6347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:27.154181 6347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:27.154227 6347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:27.154256 6347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:43:27.155915 6347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:45455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:44805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:27.156198 6347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:27.156446 6347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:27.161854 6352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:27.161834 6353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:27.161993 6347 server_base.cc:1034] running on GCE node
W20260928 04:43:27.162109 6355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:27.162335 6347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:27.162647 6347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:27.163813 6347 hybrid_clock.cc:648] HybridClock initialized: now 1790570607163767 us; error 45 us; skew 500 ppm
Sep 28 04:43:27 dist-test-slave-j5c3 krb5kdc[5975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570607, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:27.166631 6347 init.cc:377] Logged in from keytab as kudu/127.1.226.195@KRBTEST.COM (short username kudu)
I20260928 04:43:27.166965 6347 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:27.167011 6347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:27.167107 6347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:27.168227 6347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-2/data/instance:
uuid: "1be0ec8f481347a0b32ced764a2f1210"
format_stamp: "Formatted at 2026-09-28 04:43:27 on dist-test-slave-j5c3"
server_key: "5ace779e713eba82517cc9f218d35400"
server_key_iv: "b2edf903351394141d129e4181451559"
server_key_version: "encryptionkey@0"
I20260928 04:43:27.168567 6347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance:
uuid: "1be0ec8f481347a0b32ced764a2f1210"
format_stamp: "Formatted at 2026-09-28 04:43:27 on dist-test-slave-j5c3"
server_key: "5ace779e713eba82517cc9f218d35400"
server_key_iv: "b2edf903351394141d129e4181451559"
server_key_version: "encryptionkey@0"
I20260928 04:43:27.169880 6347 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:43:27.170639 6361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:27.170811 6347 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:43:27.170894 6347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "1be0ec8f481347a0b32ced764a2f1210"
format_stamp: "Formatted at 2026-09-28 04:43:27 on dist-test-slave-j5c3"
server_key: "5ace779e713eba82517cc9f218d35400"
server_key_iv: "b2edf903351394141d129e4181451559"
server_key_version: "encryptionkey@0"
I20260928 04:43:27.170949 6347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:27.178889 6347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:27.180855 6347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:27.180967 6347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:27.181147 6347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:27.181493 6347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:27.181551 6347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:27.181599 6347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:27.181627 6347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:27.187116 6347 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:45387
I20260928 04:43:27.187202 6474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:45387 every 8 connection(s)
I20260928 04:43:27.187482 6347 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
I20260928 04:43:27.187917 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 6347
I20260928 04:43:27.188041 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance
I20260928 04:43:27.188306 1931 external_mini_cluster.cc:1428] Setting key 70e45db45b1490a87b56e3d832f97e2a
Sep 28 04:43:27 dist-test-slave-j5c3 krb5kdc[5975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570607, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:27.198016 6059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.189279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:36461 (local address 127.1.226.254:44805)
0928 04:43:27.189324 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:27.189326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:27.189574 (+ 248us) server_negotiation.cc:407] Connection header received
0928 04:43:27.190117 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:27.190118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:27.190130 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:27.190178 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:27.191093 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.191721 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.192997 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.193185 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.193244 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:27.195309 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:27.195310 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:27.195310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:27.195317 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:27.196683 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:27.197045 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:27.197046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:27.197046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:27.197095 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:27.197331 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:27.197332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:27.197333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:27.197470 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:27.197538 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:27.197875 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 04:43:27.197950 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:43:27.198259 6477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.189287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44805 (local address 127.1.226.195:36461)
0928 04:43:27.189463 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:27.189499 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:27.190015 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:27.190244 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:27.190248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:27.190521 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:27.190998 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:27.191003 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.191839 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.191841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:27.192915 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:27.192918 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.193266 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.193267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:27.193369 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:27.193842 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:27.193850 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:27.195221 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:27.196758 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:27.196764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:27.196765 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:27.196982 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:27.197155 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:27.197156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:27.197156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:27.197251 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:27.197573 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:27.197580 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:27.197738 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 04:43:27.197844 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 04:43:27.197998 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":100,"threads_started":1}
I20260928 04:43:27.198815 6475 heartbeater.cc:347] Connected to a master server at 127.1.226.254:44805
I20260928 04:43:27.198935 6475 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:27.199110 6475 heartbeater.cc:511] Master 127.1.226.254:44805 requested a full tablet report, sending...
I20260928 04:43:27.199543 6019 ts_manager.cc:194] Registered new tserver with Master: 1be0ec8f481347a0b32ced764a2f1210 (127.1.226.195:45387)
I20260928 04:43:27.200058 6019 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.195@KRBTEST.COM'} at 127.1.226.195:36461
I20260928 04:43:27.200378 1931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:43:27.207706 6059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.201632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54774 (local address 127.1.226.254:44805)
0928 04:43:27.201676 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:27.201678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:27.201819 (+ 141us) server_negotiation.cc:407] Connection header received
0928 04:43:27.201840 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:27.201840 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:27.201849 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:27.201899 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:27.202386 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.202855 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.203553 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.203727 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.203775 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:27.205006 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:27.205007 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:27.205007 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:27.205013 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:27.206525 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:27.206894 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:27.206895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:27.206895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:27.206950 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:27.207128 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:27.207129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:27.207129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:27.207279 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:27.207367 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:27.207458 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 04:43:27.207519 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 04:43:27.209668 6018 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:43:27.211238 6018 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:43:27.218321 6490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.213968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57042 (local address 127.1.226.195:45387)
0928 04:43:27.214152 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:27.214158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:27.214168 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:43:27.214218 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:27.214220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:27.214233 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:27.214320 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:27.214505 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.215677 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.217215 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.217945 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.218009 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:27.218053 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:27.218195 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 04:43:27.218241 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":93,"threads_started":1}
I20260928 04:43:27.218346 6488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.213731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:45387 (local address 127.0.0.1:57042)
0928 04:43:27.214035 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:27.214044 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:27.214091 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:27.214352 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:27.214355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:27.214357 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:27.214422 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:27.214424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.215769 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.215770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:27.217121 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:27.217125 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.218026 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.218027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:27.218079 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:27.218081 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:43:27.218168 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 04:43:27.218260 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":88,"thread_start_us":94,"threads_started":1}
I20260928 04:43:27.218439 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.213985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43508 (local address 127.1.226.193:33445)
0928 04:43:27.214154 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:27.214167 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:27.214553 (+ 386us) server_negotiation.cc:407] Connection header received
0928 04:43:27.214660 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:27.214663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:27.214677 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:27.214754 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:27.214863 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.215873 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.217073 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.217850 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.217937 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:27.218005 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:27.218308 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 04:43:27.218348 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":74,"threads_started":1}
I20260928 04:43:27.218737 6487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.213746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:33445 (local address 127.0.0.1:43508)
0928 04:43:27.214474 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:27.214482 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:27.214550 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:27.214762 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:27.214762 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:27.214764 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:27.214804 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:27.214805 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.215966 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.215967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:27.216985 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:27.216988 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.217950 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.217951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:27.218061 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:27.218079 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 04:43:27.218168 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 04:43:27.218259 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"mutex_wait_us":5,"thread_start_us":92,"threads_started":1}
I20260928 04:43:27.219450 6143 tablet_service.cc:1467] Processing CreateTablet for tablet 995523fa37f44dd88cd9e800c60fde17 (DEFAULT_TABLE table=test-table [id=9aedf0e6205043ef97f213ffc0d7f5cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:27.219449 6409 tablet_service.cc:1467] Processing CreateTablet for tablet 995523fa37f44dd88cd9e800c60fde17 (DEFAULT_TABLE table=test-table [id=9aedf0e6205043ef97f213ffc0d7f5cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:27.219799 6143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 995523fa37f44dd88cd9e800c60fde17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:27.219802 6409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 995523fa37f44dd88cd9e800c60fde17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:27.220474 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.213676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:41327 (local address 127.0.0.1:43248)
0928 04:43:27.213796 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:27.213837 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:27.213908 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:27.214649 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:27.214650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:27.214652 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:27.214703 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:27.214704 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.215684 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.215684 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:27.217127 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:27.217128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.220233 (+ 3105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.220234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:27.220300 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:27.220304 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:43:27.220359 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 04:43:27.220400 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":63,"thread_start_us":74,"threads_started":1}
I20260928 04:43:27.220845 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.213870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43248 (local address 127.1.226.194:41327)
0928 04:43:27.214230 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:27.214233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:27.214241 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:43:27.214274 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:27.214275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:27.214287 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:27.214805 (+ 518us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:27.214884 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.215581 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.218096 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.218987 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.219065 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:27.219121 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:27.220726 (+ 1605us) server_negotiation.cc:299] Negotiation successful
0928 04:43:27.220772 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":85,"threads_started":1}
I20260928 04:43:27.223230 6492 tablet_bootstrap.cc:492] T 995523fa37f44dd88cd9e800c60fde17 P 5ce63c1c94854c668a0698b273f429bc: Bootstrap starting.
I20260928 04:43:27.223477 6493 tablet_bootstrap.cc:492] T 995523fa37f44dd88cd9e800c60fde17 P 1be0ec8f481347a0b32ced764a2f1210: Bootstrap starting.
I20260928 04:43:27.224078 6493 tablet_bootstrap.cc:654] T 995523fa37f44dd88cd9e800c60fde17 P 1be0ec8f481347a0b32ced764a2f1210: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:27.224295 6492 tablet_bootstrap.cc:654] T 995523fa37f44dd88cd9e800c60fde17 P 5ce63c1c94854c668a0698b273f429bc: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:27.224661 6493 log.cc:824] T 995523fa37f44dd88cd9e800c60fde17 P 1be0ec8f481347a0b32ced764a2f1210: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:27.225309 6492 log.cc:824] T 995523fa37f44dd88cd9e800c60fde17 P 5ce63c1c94854c668a0698b273f429bc: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:27.225569 6493 tablet_bootstrap.cc:492] T 995523fa37f44dd88cd9e800c60fde17 P 1be0ec8f481347a0b32ced764a2f1210: No bootstrap required, opened a new log
I20260928 04:43:27.225622 6493 ts_tablet_manager.cc:1397] T 995523fa37f44dd88cd9e800c60fde17 P 1be0ec8f481347a0b32ced764a2f1210: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 04:43:27.226270 6492 tablet_bootstrap.cc:492] T 995523fa37f44dd88cd9e800c60fde17 P 5ce63c1c94854c668a0698b273f429bc: No bootstrap required, opened a new log
I20260928 04:43:27.226346 6492 ts_tablet_manager.cc:1397] T 995523fa37f44dd88cd9e800c60fde17 P 5ce63c1c94854c668a0698b273f429bc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 04:43:27.227378 6493 raft_consensus.cc:348] T 995523fa37f44dd88cd9e800c60fde17 P 1be0ec8f481347a0b32ced764a2f1210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42972670eb0d422db52d9e0af67c9fa9" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 41327 } } peers { permanent_uuid: "5ce63c1c94854c668a0698b273f429bc" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 33445 } } peers { permanent_uuid: "1be0ec8f481347a0b32ced764a2f1210" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45387 } }
I20260928 04:43:27.227535 6493 raft_consensus.cc:374] T 995523fa37f44dd88cd9e800c60fde17 P 1be0ec8f481347a0b32ced764a2f1210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:27.227562 6493 raft_consensus.cc:729] T 995523fa37f44dd88cd9e800c60fde17 P 1be0ec8f481347a0b32ced764a2f1210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1be0ec8f481347a0b32ced764a2f1210, State: Initialized, Role: FOLLOWER
I20260928 04:43:27.227701 6493 consensus_queue.cc:260] T 995523fa37f44dd88cd9e800c60fde17 P 1be0ec8f481347a0b32ced764a2f1210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42972670eb0d422db52d9e0af67c9fa9" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 41327 } } peers { permanent_uuid: "5ce63c1c94854c668a0698b273f429bc" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 33445 } } peers { permanent_uuid: "1be0ec8f481347a0b32ced764a2f1210" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45387 } }
I20260928 04:43:27.227954 6493 ts_tablet_manager.cc:1428] T 995523fa37f44dd88cd9e800c60fde17 P 1be0ec8f481347a0b32ced764a2f1210: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 04:43:27.227986 6475 heartbeater.cc:503] Master 127.1.226.254:44805 was elected leader, sending a full tablet report...
I20260928 04:43:27.227986 6492 raft_consensus.cc:348] T 995523fa37f44dd88cd9e800c60fde17 P 5ce63c1c94854c668a0698b273f429bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42972670eb0d422db52d9e0af67c9fa9" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 41327 } } peers { permanent_uuid: "5ce63c1c94854c668a0698b273f429bc" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 33445 } } peers { permanent_uuid: "1be0ec8f481347a0b32ced764a2f1210" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45387 } }
I20260928 04:43:27.232746 6276 tablet_service.cc:1467] Processing CreateTablet for tablet 995523fa37f44dd88cd9e800c60fde17 (DEFAULT_TABLE table=test-table [id=9aedf0e6205043ef97f213ffc0d7f5cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:27.233131 6276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 995523fa37f44dd88cd9e800c60fde17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:27.228206 6492 raft_consensus.cc:374] T 995523fa37f44dd88cd9e800c60fde17 P 5ce63c1c94854c668a0698b273f429bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:27.233963 6492 raft_consensus.cc:729] T 995523fa37f44dd88cd9e800c60fde17 P 5ce63c1c94854c668a0698b273f429bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ce63c1c94854c668a0698b273f429bc, State: Initialized, Role: FOLLOWER
I20260928 04:43:27.234107 6492 consensus_queue.cc:260] T 995523fa37f44dd88cd9e800c60fde17 P 5ce63c1c94854c668a0698b273f429bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42972670eb0d422db52d9e0af67c9fa9" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 41327 } } peers { permanent_uuid: "5ce63c1c94854c668a0698b273f429bc" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 33445 } } peers { permanent_uuid: "1be0ec8f481347a0b32ced764a2f1210" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45387 } }
I20260928 04:43:27.234455 6492 ts_tablet_manager.cc:1428] T 995523fa37f44dd88cd9e800c60fde17 P 5ce63c1c94854c668a0698b273f429bc: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20260928 04:43:27.234632 6209 heartbeater.cc:503] Master 127.1.226.254:44805 was elected leader, sending a full tablet report...
I20260928 04:43:27.238355 6498 tablet_bootstrap.cc:492] T 995523fa37f44dd88cd9e800c60fde17 P 42972670eb0d422db52d9e0af67c9fa9: Bootstrap starting.
I20260928 04:43:27.238974 6498 tablet_bootstrap.cc:654] T 995523fa37f44dd88cd9e800c60fde17 P 42972670eb0d422db52d9e0af67c9fa9: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:27.239523 6498 log.cc:824] T 995523fa37f44dd88cd9e800c60fde17 P 42972670eb0d422db52d9e0af67c9fa9: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:27.240548 6498 tablet_bootstrap.cc:492] T 995523fa37f44dd88cd9e800c60fde17 P 42972670eb0d422db52d9e0af67c9fa9: No bootstrap required, opened a new log
I20260928 04:43:27.240636 6498 ts_tablet_manager.cc:1397] T 995523fa37f44dd88cd9e800c60fde17 P 42972670eb0d422db52d9e0af67c9fa9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:43:27.242544 6497 raft_consensus.cc:482] T 995523fa37f44dd88cd9e800c60fde17 P 5ce63c1c94854c668a0698b273f429bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:43:27.242666 6497 raft_consensus.cc:504] T 995523fa37f44dd88cd9e800c60fde17 P 5ce63c1c94854c668a0698b273f429bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42972670eb0d422db52d9e0af67c9fa9" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 41327 } } peers { permanent_uuid: "5ce63c1c94854c668a0698b273f429bc" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 33445 } } peers { permanent_uuid: "1be0ec8f481347a0b32ced764a2f1210" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45387 } }
I20260928 04:43:27.243062 6497 leader_election.cc:288] T 995523fa37f44dd88cd9e800c60fde17 P 5ce63c1c94854c668a0698b273f429bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42972670eb0d422db52d9e0af67c9fa9 (127.1.226.194:41327), 1be0ec8f481347a0b32ced764a2f1210 (127.1.226.195:45387)
I20260928 04:43:27.243034 6498 raft_consensus.cc:348] T 995523fa37f44dd88cd9e800c60fde17 P 42972670eb0d422db52d9e0af67c9fa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42972670eb0d422db52d9e0af67c9fa9" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 41327 } } peers { permanent_uuid: "5ce63c1c94854c668a0698b273f429bc" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 33445 } } peers { permanent_uuid: "1be0ec8f481347a0b32ced764a2f1210" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45387 } }
I20260928 04:43:27.243217 6498 raft_consensus.cc:374] T 995523fa37f44dd88cd9e800c60fde17 P 42972670eb0d422db52d9e0af67c9fa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:27.243266 6498 raft_consensus.cc:729] T 995523fa37f44dd88cd9e800c60fde17 P 42972670eb0d422db52d9e0af67c9fa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42972670eb0d422db52d9e0af67c9fa9, State: Initialized, Role: FOLLOWER
I20260928 04:43:27.243716 6498 consensus_queue.cc:260] T 995523fa37f44dd88cd9e800c60fde17 P 42972670eb0d422db52d9e0af67c9fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42972670eb0d422db52d9e0af67c9fa9" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 41327 } } peers { permanent_uuid: "5ce63c1c94854c668a0698b273f429bc" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 33445 } } peers { permanent_uuid: "1be0ec8f481347a0b32ced764a2f1210" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45387 } }
I20260928 04:43:27.244036 6498 ts_tablet_manager.cc:1428] T 995523fa37f44dd88cd9e800c60fde17 P 42972670eb0d422db52d9e0af67c9fa9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:43:27.244213 6342 heartbeater.cc:503] Master 127.1.226.254:44805 was elected leader, sending a full tablet report...
I20260928 04:43:27.246798 6211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.243214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:45387 (local address 127.1.226.193:55323)
0928 04:43:27.243265 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:27.243281 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:27.243347 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:27.243546 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:27.243547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:27.243549 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:27.243623 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:27.243625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.244564 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.244565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:27.245744 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:27.245747 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.246529 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.246530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:27.246611 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:27.246614 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:43:27.246668 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 04:43:27.246713 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 04:43:27.246806 6490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.243347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:55323 (local address 127.1.226.195:45387)
0928 04:43:27.243400 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:27.243402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:27.243412 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:43:27.243453 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:27.243454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:27.243465 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:27.243541 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:27.243713 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.244444 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.245843 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.246424 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.246521 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:27.246532 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:27.246677 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 04:43:27.246734 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 04:43:27.246860 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.243314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:49155 (local address 127.1.226.194:41327)
0928 04:43:27.243356 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:27.243358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:27.243598 (+ 240us) server_negotiation.cc:407] Connection header received
0928 04:43:27.243700 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:27.243701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:27.243711 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:27.243740 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:27.243958 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.244666 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.245832 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.246497 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.246565 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:27.246572 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:27.246763 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 04:43:27.246796 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:43:27.247102 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.243224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:41327 (local address 127.1.226.193:49155)
0928 04:43:27.243533 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:27.243543 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:27.243626 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:27.243805 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:27.243806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:27.243807 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:27.243877 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:27.243879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.244739 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.244740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:27.245743 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:27.245745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.246601 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.246602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:27.246674 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:27.246677 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:43:27.246960 (+ 283us) client_negotiation.cc:240] Negotiation successful
0928 04:43:27.247009 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":146,"thread_start_us":113,"threads_started":1}
I20260928 04:43:27.247426 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "995523fa37f44dd88cd9e800c60fde17" candidate_uuid: "5ce63c1c94854c668a0698b273f429bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42972670eb0d422db52d9e0af67c9fa9" is_pre_election: true
I20260928 04:43:27.247516 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "995523fa37f44dd88cd9e800c60fde17" candidate_uuid: "5ce63c1c94854c668a0698b273f429bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1be0ec8f481347a0b32ced764a2f1210" is_pre_election: true
I20260928 04:43:27.247593 6296 raft_consensus.cc:2454] T 995523fa37f44dd88cd9e800c60fde17 P 42972670eb0d422db52d9e0af67c9fa9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ce63c1c94854c668a0698b273f429bc in term 0.
I20260928 04:43:27.247699 6429 raft_consensus.cc:2454] T 995523fa37f44dd88cd9e800c60fde17 P 1be0ec8f481347a0b32ced764a2f1210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ce63c1c94854c668a0698b273f429bc in term 0.
I20260928 04:43:27.247853 6098 leader_election.cc:302] T 995523fa37f44dd88cd9e800c60fde17 P 5ce63c1c94854c668a0698b273f429bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42972670eb0d422db52d9e0af67c9fa9, 5ce63c1c94854c668a0698b273f429bc; no voters:
I20260928 04:43:27.248013 6497 raft_consensus.cc:2781] T 995523fa37f44dd88cd9e800c60fde17 P 5ce63c1c94854c668a0698b273f429bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:43:27.248073 6497 raft_consensus.cc:482] T 995523fa37f44dd88cd9e800c60fde17 P 5ce63c1c94854c668a0698b273f429bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:43:27.248107 6497 raft_consensus.cc:3037] T 995523fa37f44dd88cd9e800c60fde17 P 5ce63c1c94854c668a0698b273f429bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:27.248847 6497 raft_consensus.cc:504] T 995523fa37f44dd88cd9e800c60fde17 P 5ce63c1c94854c668a0698b273f429bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42972670eb0d422db52d9e0af67c9fa9" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 41327 } } peers { permanent_uuid: "5ce63c1c94854c668a0698b273f429bc" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 33445 } } peers { permanent_uuid: "1be0ec8f481347a0b32ced764a2f1210" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45387 } }
I20260928 04:43:27.249008 6497 leader_election.cc:288] T 995523fa37f44dd88cd9e800c60fde17 P 5ce63c1c94854c668a0698b273f429bc [CANDIDATE]: Term 1 election: Requested vote from peers 42972670eb0d422db52d9e0af67c9fa9 (127.1.226.194:41327), 1be0ec8f481347a0b32ced764a2f1210 (127.1.226.195:45387)
I20260928 04:43:27.249145 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "995523fa37f44dd88cd9e800c60fde17" candidate_uuid: "5ce63c1c94854c668a0698b273f429bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1be0ec8f481347a0b32ced764a2f1210"
I20260928 04:43:27.249181 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "995523fa37f44dd88cd9e800c60fde17" candidate_uuid: "5ce63c1c94854c668a0698b273f429bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42972670eb0d422db52d9e0af67c9fa9"
I20260928 04:43:27.249248 6296 raft_consensus.cc:3037] T 995523fa37f44dd88cd9e800c60fde17 P 42972670eb0d422db52d9e0af67c9fa9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:27.249260 6429 raft_consensus.cc:3037] T 995523fa37f44dd88cd9e800c60fde17 P 1be0ec8f481347a0b32ced764a2f1210 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:27.249943 6429 raft_consensus.cc:2454] T 995523fa37f44dd88cd9e800c60fde17 P 1be0ec8f481347a0b32ced764a2f1210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce63c1c94854c668a0698b273f429bc in term 1.
I20260928 04:43:27.249953 6296 raft_consensus.cc:2454] T 995523fa37f44dd88cd9e800c60fde17 P 42972670eb0d422db52d9e0af67c9fa9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce63c1c94854c668a0698b273f429bc in term 1.
I20260928 04:43:27.250108 6096 leader_election.cc:302] T 995523fa37f44dd88cd9e800c60fde17 P 5ce63c1c94854c668a0698b273f429bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1be0ec8f481347a0b32ced764a2f1210, 5ce63c1c94854c668a0698b273f429bc; no voters:
I20260928 04:43:27.250221 6497 raft_consensus.cc:2781] T 995523fa37f44dd88cd9e800c60fde17 P 5ce63c1c94854c668a0698b273f429bc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:27.250409 6497 raft_consensus.cc:686] T 995523fa37f44dd88cd9e800c60fde17 P 5ce63c1c94854c668a0698b273f429bc [term 1 LEADER]: Becoming Leader. State: Replica: 5ce63c1c94854c668a0698b273f429bc, State: Running, Role: LEADER
I20260928 04:43:27.250514 6497 consensus_queue.cc:237] T 995523fa37f44dd88cd9e800c60fde17 P 5ce63c1c94854c668a0698b273f429bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42972670eb0d422db52d9e0af67c9fa9" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 41327 } } peers { permanent_uuid: "5ce63c1c94854c668a0698b273f429bc" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 33445 } } peers { permanent_uuid: "1be0ec8f481347a0b32ced764a2f1210" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45387 } }
I20260928 04:43:27.251381 6018 catalog_manager.cc:5495] T 995523fa37f44dd88cd9e800c60fde17 P 5ce63c1c94854c668a0698b273f429bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ce63c1c94854c668a0698b273f429bc (127.1.226.193). New cstate: current_term: 1 leader_uuid: "5ce63c1c94854c668a0698b273f429bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42972670eb0d422db52d9e0af67c9fa9" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 41327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ce63c1c94854c668a0698b273f429bc" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 33445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1be0ec8f481347a0b32ced764a2f1210" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45387 } health_report { overall_health: UNKNOWN } } }
I20260928 04:43:27.256577 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.254315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43514 (local address 127.1.226.193:33445)
0928 04:43:27.254370 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:27.254372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:27.254380 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:43:27.254417 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:27.254418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:27.254432 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:27.254501 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:43:27.254653 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.255165 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.255883 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:27.256056 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:27.256104 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:27.256302 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:43:27.256395 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:43:27.256454 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:27.256489 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 04:43:27.256506 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 04:43:27.258345 6296 raft_consensus.cc:1261] T 995523fa37f44dd88cd9e800c60fde17 P 42972670eb0d422db52d9e0af67c9fa9 [term 1 FOLLOWER]: Refusing update from remote peer 5ce63c1c94854c668a0698b273f429bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:27.258378 6429 raft_consensus.cc:1261] T 995523fa37f44dd88cd9e800c60fde17 P 1be0ec8f481347a0b32ced764a2f1210 [term 1 FOLLOWER]: Refusing update from remote peer 5ce63c1c94854c668a0698b273f429bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:27.258733 6502 consensus_queue.cc:1035] T 995523fa37f44dd88cd9e800c60fde17 P 5ce63c1c94854c668a0698b273f429bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1be0ec8f481347a0b32ced764a2f1210" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:27.258864 6497 consensus_queue.cc:1035] T 995523fa37f44dd88cd9e800c60fde17 P 5ce63c1c94854c668a0698b273f429bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "42972670eb0d422db52d9e0af67c9fa9" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 41327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:27.261351 6505 mvcc.cc:205] Tried to move back new op lower bound from 7334177207327404032 to 7334177207298830336. Current Snapshot: MvccSnapshot[applied={T|T < 7334177207327404032}]
I20260928 04:43:27.261942 6507 mvcc.cc:205] Tried to move back new op lower bound from 7334177207327404032 to 7334177207298830336. Current Snapshot: MvccSnapshot[applied={T|T < 7334177207327404032}]
I20260928 04:43:27.262270 6506 mvcc.cc:205] Tried to move back new op lower bound from 7334177207327404032 to 7334177207298830336. Current Snapshot: MvccSnapshot[applied={T|T < 7334177207327404032}]
I20260928 04:43:27.263631 6019 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:43:27.263743 6019 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:43:27.264554 6019 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3a10584a84643b69bb1db59ff1a609f: Sending DeleteTablet for 3 replicas of tablet 995523fa37f44dd88cd9e800c60fde17
I20260928 04:43:27.264887 6143 tablet_service.cc:1514] Processing DeleteTablet for tablet 995523fa37f44dd88cd9e800c60fde17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:27 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:43508
I20260928 04:43:27.264906 6409 tablet_service.cc:1514] Processing DeleteTablet for tablet 995523fa37f44dd88cd9e800c60fde17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:27 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:57042
I20260928 04:43:27.264926 6276 tablet_service.cc:1514] Processing DeleteTablet for tablet 995523fa37f44dd88cd9e800c60fde17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:27 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:43248
I20260928 04:43:27.265126 6515 tablet_replica.cc:330] T 995523fa37f44dd88cd9e800c60fde17 P 5ce63c1c94854c668a0698b273f429bc: stopping tablet replica
I20260928 04:43:27.265223 6515 raft_consensus.cc:2229] T 995523fa37f44dd88cd9e800c60fde17 P 5ce63c1c94854c668a0698b273f429bc [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:43:27.265322 6515 raft_consensus.cc:2258] T 995523fa37f44dd88cd9e800c60fde17 P 5ce63c1c94854c668a0698b273f429bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:27.265303 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 6081
I20260928 04:43:27.265463 6517 tablet_replica.cc:330] T 995523fa37f44dd88cd9e800c60fde17 P 42972670eb0d422db52d9e0af67c9fa9: stopping tablet replica
I20260928 04:43:27.265554 6517 raft_consensus.cc:2229] T 995523fa37f44dd88cd9e800c60fde17 P 42972670eb0d422db52d9e0af67c9fa9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:43:27.265640 6517 raft_consensus.cc:2258] T 995523fa37f44dd88cd9e800c60fde17 P 42972670eb0d422db52d9e0af67c9fa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:27.266106 6516 tablet_replica.cc:330] T 995523fa37f44dd88cd9e800c60fde17 P 1be0ec8f481347a0b32ced764a2f1210: stopping tablet replica
I20260928 04:43:27.266220 6516 raft_consensus.cc:2229] T 995523fa37f44dd88cd9e800c60fde17 P 1be0ec8f481347a0b32ced764a2f1210 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:43:27.266281 6517 ts_tablet_manager.cc:1905] T 995523fa37f44dd88cd9e800c60fde17 P 42972670eb0d422db52d9e0af67c9fa9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:27.266324 6516 raft_consensus.cc:2258] T 995523fa37f44dd88cd9e800c60fde17 P 1be0ec8f481347a0b32ced764a2f1210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:27.266691 6516 ts_tablet_manager.cc:1905] T 995523fa37f44dd88cd9e800c60fde17 P 1be0ec8f481347a0b32ced764a2f1210: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:27.271024 6516 ts_tablet_manager.cc:1918] T 995523fa37f44dd88cd9e800c60fde17 P 1be0ec8f481347a0b32ced764a2f1210: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:43:27.271130 6516 log.cc:1196] T 995523fa37f44dd88cd9e800c60fde17 P 1be0ec8f481347a0b32ced764a2f1210: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-2/wal/wals/995523fa37f44dd88cd9e800c60fde17
I20260928 04:43:27.271431 6516 ts_tablet_manager.cc:1939] T 995523fa37f44dd88cd9e800c60fde17 P 1be0ec8f481347a0b32ced764a2f1210: Deleting consensus metadata
I20260928 04:43:27.271982 6003 catalog_manager.cc:4841] TS 1be0ec8f481347a0b32ced764a2f1210 (127.1.226.195:45387): tablet 995523fa37f44dd88cd9e800c60fde17 (table test-table [id=9aedf0e6205043ef97f213ffc0d7f5cb]) successfully deleted
W20260928 04:43:27.272351 6006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.226.193:33445 (error 108)
I20260928 04:43:27.272576 6517 ts_tablet_manager.cc:1918] T 995523fa37f44dd88cd9e800c60fde17 P 42972670eb0d422db52d9e0af67c9fa9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:43:27.272657 6517 log.cc:1196] T 995523fa37f44dd88cd9e800c60fde17 P 42972670eb0d422db52d9e0af67c9fa9: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570585693787-1931-0/minicluster-data/ts-1/wal/wals/995523fa37f44dd88cd9e800c60fde17
I20260928 04:43:27.272709 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 6214
I20260928 04:43:27.272930 6517 ts_tablet_manager.cc:1939] T 995523fa37f44dd88cd9e800c60fde17 P 42972670eb0d422db52d9e0af67c9fa9: Deleting consensus metadata
I20260928 04:43:27.276571 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.276371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:33445 (local address 127.0.0.1:43516)
0928 04:43:27.276475 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:27.276535 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.193:33445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 04:43:27.276829 6006 catalog_manager.cc:4584] TS 5ce63c1c94854c668a0698b273f429bc (127.1.226.193:33445): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 995523fa37f44dd88cd9e800c60fde17: Network error: Client connection negotiation failed: client connection to 127.1.226.193:33445: connect: Connection refused (error 111)
I20260928 04:43:27.277952 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.277820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:41327 (local address 127.0.0.1:43260)
0928 04:43:27.277891 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:27.277918 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.194:41327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 04:43:27.277942 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 6347
I20260928 04:43:27.282719 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 5988
2026-09-28T04:43:27Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1929 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T04:43:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:43:27Z Disabled control of system clock
I20260928 04:43:27.310390 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:42177
--webserver_interface=127.1.226.254
--webserver_port=0
--builtin_ntp_servers=127.1.226.212:34055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:42177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 04:43:27.391690 6522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:27.391953 6522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:27.391979 6522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:27.393317 6522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:43:27.393368 6522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:27.393381 6522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:27.393394 6522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:43:27.393406 6522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:43:27.394842 6522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:34055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:42177
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:42177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:27.395126 6522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:27.395458 6522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:27.401147 6528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:27.401374 6522 server_base.cc:1034] running on GCE node
W20260928 04:43:27.401199 6530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:27.401177 6527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:27.401841 6522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:27.402156 6522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:27.403366 6522 hybrid_clock.cc:648] HybridClock initialized: now 1790570607403351 us; error 33 us; skew 500 ppm
I20260928 04:43:27.404712 6522 webserver.cc:460] Webserver started at http://127.1.226.254:35583/ using document root <none> and password file <none>
I20260928 04:43:27.404932 6522 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:27.404974 6522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:27.405066 6522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:27.406113 6522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "b15d7b7be37645538c152780f408435d"
format_stamp: "Formatted at 2026-09-28 04:43:27 on dist-test-slave-j5c3"
server_key: "6baea0af7eb9c4a54f1665850350a05a"
server_key_iv: "e8d8398a90d73d5818a5b1d89cfceaf2"
server_key_version: "encryptionkey@0"
I20260928 04:43:27.406418 6522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "b15d7b7be37645538c152780f408435d"
format_stamp: "Formatted at 2026-09-28 04:43:27 on dist-test-slave-j5c3"
server_key: "6baea0af7eb9c4a54f1665850350a05a"
server_key_iv: "e8d8398a90d73d5818a5b1d89cfceaf2"
server_key_version: "encryptionkey@0"
I20260928 04:43:27.407634 6522 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:43:27.408396 6536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:27.408618 6522 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:27.408776 6522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "b15d7b7be37645538c152780f408435d"
format_stamp: "Formatted at 2026-09-28 04:43:27 on dist-test-slave-j5c3"
server_key: "6baea0af7eb9c4a54f1665850350a05a"
server_key_iv: "e8d8398a90d73d5818a5b1d89cfceaf2"
server_key_version: "encryptionkey@0"
I20260928 04:43:27.408862 6522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:27.412122 6522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:27.412437 6522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:27.412587 6522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:27.416404 6522 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:42177
I20260928 04:43:27.416432 6588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:42177 every 8 connection(s)
I20260928 04:43:27.416797 6522 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:43:27.417456 6589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:27.420300 6589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b15d7b7be37645538c152780f408435d: Bootstrap starting.
I20260928 04:43:27.420883 6589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b15d7b7be37645538c152780f408435d: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:27.421175 6589 log.cc:824] T 00000000000000000000000000000000 P b15d7b7be37645538c152780f408435d: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:27.421852 6589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b15d7b7be37645538c152780f408435d: No bootstrap required, opened a new log
I20260928 04:43:27.423436 6589 raft_consensus.cc:348] T 00000000000000000000000000000000 P b15d7b7be37645538c152780f408435d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15d7b7be37645538c152780f408435d" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42177 } }
I20260928 04:43:27.423575 6589 raft_consensus.cc:374] T 00000000000000000000000000000000 P b15d7b7be37645538c152780f408435d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:27.423641 6589 raft_consensus.cc:729] T 00000000000000000000000000000000 P b15d7b7be37645538c152780f408435d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b15d7b7be37645538c152780f408435d, State: Initialized, Role: FOLLOWER
I20260928 04:43:27.423790 6589 consensus_queue.cc:260] T 00000000000000000000000000000000 P b15d7b7be37645538c152780f408435d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15d7b7be37645538c152780f408435d" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42177 } }
I20260928 04:43:27.423852 6589 raft_consensus.cc:388] T 00000000000000000000000000000000 P b15d7b7be37645538c152780f408435d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:43:27.423872 6589 raft_consensus.cc:482] T 00000000000000000000000000000000 P b15d7b7be37645538c152780f408435d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:43:27.423938 6589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b15d7b7be37645538c152780f408435d [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:27.424449 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 6522
I20260928 04:43:27.424556 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:43:27.424755 1931 external_mini_cluster.cc:1428] Setting key 41848a855493ee8f653c4faf297a8a70
I20260928 04:43:27.424749 6589 raft_consensus.cc:504] T 00000000000000000000000000000000 P b15d7b7be37645538c152780f408435d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15d7b7be37645538c152780f408435d" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42177 } }
I20260928 04:43:27.424847 6589 leader_election.cc:302] T 00000000000000000000000000000000 P b15d7b7be37645538c152780f408435d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b15d7b7be37645538c152780f408435d; no voters:
I20260928 04:43:27.425029 6589 leader_election.cc:288] T 00000000000000000000000000000000 P b15d7b7be37645538c152780f408435d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:43:27.425064 6592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b15d7b7be37645538c152780f408435d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:27.425257 6592 raft_consensus.cc:686] T 00000000000000000000000000000000 P b15d7b7be37645538c152780f408435d [term 1 LEADER]: Becoming Leader. State: Replica: b15d7b7be37645538c152780f408435d, State: Running, Role: LEADER
I20260928 04:43:27.425309 6589 sys_catalog.cc:564] T 00000000000000000000000000000000 P b15d7b7be37645538c152780f408435d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:43:27.425357 6592 consensus_queue.cc:237] T 00000000000000000000000000000000 P b15d7b7be37645538c152780f408435d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15d7b7be37645538c152780f408435d" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42177 } }
I20260928 04:43:27.425697 6592 sys_catalog.cc:455] T 00000000000000000000000000000000 P b15d7b7be37645538c152780f408435d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b15d7b7be37645538c152780f408435d. Latest consensus state: current_term: 1 leader_uuid: "b15d7b7be37645538c152780f408435d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15d7b7be37645538c152780f408435d" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42177 } } }
I20260928 04:43:27.425786 6592 sys_catalog.cc:458] T 00000000000000000000000000000000 P b15d7b7be37645538c152780f408435d [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:27.426069 6595 sys_catalog.cc:455] T 00000000000000000000000000000000 P b15d7b7be37645538c152780f408435d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b15d7b7be37645538c152780f408435d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15d7b7be37645538c152780f408435d" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42177 } } }
I20260928 04:43:27.426156 6595 sys_catalog.cc:458] T 00000000000000000000000000000000 P b15d7b7be37645538c152780f408435d [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:27.427112 6596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.425335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41246 (local address 127.1.226.254:42177)
0928 04:43:27.425550 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:27.425560 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:27.425591 (+ 31us) server_negotiation.cc:407] Connection header received
0928 04:43:27.425714 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:27.425722 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:27.425775 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:27.426236 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:27.426610 (+ 374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:27.426614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:27.426614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:43:27.426618 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:27.426660 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:43:27.426677 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:27.426782 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:27.426838 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 04:43:27.426905 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
W20260928 04:43:27.427486 6609 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b15d7b7be37645538c152780f408435d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:43:27.427536 6609 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:43:27.428200 6601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:43:27.428370 6601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:43:27.429749 6601 catalog_manager.cc:1347] Generated new cluster ID: 1cd1455c254c48bcbba91ca62fbbea10
I20260928 04:43:27.429800 6601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:43:27.460144 6601 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:43:27.460417 6601 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:43:27.464110 6601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b15d7b7be37645538c152780f408435d: Generated new TSK 0
I20260928 04:43:27.464366 6601 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:43:27.467669 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:0
--local_ip_for_outbound_sockets=127.1.226.193
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:42177
--builtin_ntp_servers=127.1.226.212:34055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 04:43:27.549602 6613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:27.549844 6613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:27.549870 6613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:27.551162 6613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:27.551209 6613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:27.551239 6613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:43:27.552771 6613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:34055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:42177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:27.553042 6613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:27.553362 6613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:27.559718 6619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:27.559770 6621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:27.559782 6613 server_base.cc:1034] running on GCE node
W20260928 04:43:27.559695 6618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:27.560222 6613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:27.560477 6613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:27.561656 6613 hybrid_clock.cc:648] HybridClock initialized: now 1790570607561648 us; error 41 us; skew 500 ppm
I20260928 04:43:27.563169 6613 webserver.cc:460] Webserver started at http://127.1.226.193:43207/ using document root <none> and password file <none>
I20260928 04:43:27.563426 6613 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:27.563496 6613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:27.563657 6613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:27.564895 6613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-0/data/instance:
uuid: "6830682b21ed40b6a9f0cb21d7f2e94c"
format_stamp: "Formatted at 2026-09-28 04:43:27 on dist-test-slave-j5c3"
server_key: "524c343389adcc325b542bbb5e0cc6a5"
server_key_iv: "27039f9db350f41c40dc0c2a5fb5a534"
server_key_version: "encryptionkey@0"
I20260928 04:43:27.565281 6613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance:
uuid: "6830682b21ed40b6a9f0cb21d7f2e94c"
format_stamp: "Formatted at 2026-09-28 04:43:27 on dist-test-slave-j5c3"
server_key: "524c343389adcc325b542bbb5e0cc6a5"
server_key_iv: "27039f9db350f41c40dc0c2a5fb5a534"
server_key_version: "encryptionkey@0"
I20260928 04:43:27.566681 6613 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:43:27.567507 6627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:27.567674 6613 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:43:27.567838 6613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "6830682b21ed40b6a9f0cb21d7f2e94c"
format_stamp: "Formatted at 2026-09-28 04:43:27 on dist-test-slave-j5c3"
server_key: "524c343389adcc325b542bbb5e0cc6a5"
server_key_iv: "27039f9db350f41c40dc0c2a5fb5a534"
server_key_version: "encryptionkey@0"
I20260928 04:43:27.567917 6613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:27.571403 6613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:27.571709 6613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:27.571856 6613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:27.572070 6613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:27.572441 6613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:27.572497 6613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:27.572527 6613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:27.572551 6613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:27.577981 6613 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:43405
I20260928 04:43:27.578105 6740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:43405 every 8 connection(s)
I20260928 04:43:27.578368 6613 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
I20260928 04:43:27.581835 6596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.579197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:53171 (local address 127.1.226.254:42177)
0928 04:43:27.579261 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:27.579264 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:27.579488 (+ 224us) server_negotiation.cc:407] Connection header received
0928 04:43:27.580044 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:27.580045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:27.580054 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:27.580113 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:27.581550 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:27.581552 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:27.581553 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:43:27.581558 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:27.581581 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:43:27.581585 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:27.581653 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:27.581739 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 04:43:27.581758 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:43:27.582051 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.579166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42177 (local address 127.1.226.193:53171)
0928 04:43:27.579353 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:27.579387 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:27.579928 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:27.580232 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:27.580236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:27.580245 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:27.581368 (+ 1123us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:43:27.581369 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:27.581399 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:43:27.581430 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:27.581650 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:27.581657 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:27.581660 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:43:27.581742 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 04:43:27.581817 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260928 04:43:27.582598 6741 heartbeater.cc:347] Connected to a master server at 127.1.226.254:42177
I20260928 04:43:27.582713 6741 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:27.582783 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 6613
I20260928 04:43:27.582847 6741 heartbeater.cc:511] Master 127.1.226.254:42177 requested a full tablet report, sending...
I20260928 04:43:27.582888 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance
I20260928 04:43:27.583081 1931 external_mini_cluster.cc:1428] Setting key 78661e19a387e618717e01917426ec8f
I20260928 04:43:27.583508 6553 ts_manager.cc:194] Registered new tserver with Master: 6830682b21ed40b6a9f0cb21d7f2e94c (127.1.226.193:43405)
I20260928 04:43:27.584367 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:0
--local_ip_for_outbound_sockets=127.1.226.194
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:42177
--builtin_ntp_servers=127.1.226.212:34055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 04:43:27.670246 6744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:27.670475 6744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:27.670499 6744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:27.671799 6744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:27.671840 6744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:27.671867 6744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:43:27.673275 6744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:34055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:42177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:27.673487 6744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:27.673739 6744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:27.678741 6749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:27.678732 6752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:27.678732 6750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:27.679068 6744 server_base.cc:1034] running on GCE node
I20260928 04:43:27.679273 6744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:27.679483 6744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:27.680624 6744 hybrid_clock.cc:648] HybridClock initialized: now 1790570607680607 us; error 33 us; skew 500 ppm
I20260928 04:43:27.681963 6744 webserver.cc:460] Webserver started at http://127.1.226.194:37575/ using document root <none> and password file <none>
I20260928 04:43:27.682163 6744 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:27.682204 6744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:27.682300 6744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:27.683329 6744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-1/data/instance:
uuid: "369f570c04e44b8282d1cc652031e93b"
format_stamp: "Formatted at 2026-09-28 04:43:27 on dist-test-slave-j5c3"
server_key: "c34b36d868a7650fc2fc6ecc7f9ab8c2"
server_key_iv: "c78fd25938dbc5037efcf451a7e6dee7"
server_key_version: "encryptionkey@0"
I20260928 04:43:27.683651 6744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance:
uuid: "369f570c04e44b8282d1cc652031e93b"
format_stamp: "Formatted at 2026-09-28 04:43:27 on dist-test-slave-j5c3"
server_key: "c34b36d868a7650fc2fc6ecc7f9ab8c2"
server_key_iv: "c78fd25938dbc5037efcf451a7e6dee7"
server_key_version: "encryptionkey@0"
I20260928 04:43:27.685006 6744 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:43:27.685801 6758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:27.686007 6744 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:27.686136 6744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "369f570c04e44b8282d1cc652031e93b"
format_stamp: "Formatted at 2026-09-28 04:43:27 on dist-test-slave-j5c3"
server_key: "c34b36d868a7650fc2fc6ecc7f9ab8c2"
server_key_iv: "c78fd25938dbc5037efcf451a7e6dee7"
server_key_version: "encryptionkey@0"
I20260928 04:43:27.686199 6744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:27.689690 6744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:27.689915 6744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:27.690012 6744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:27.690184 6744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:27.690438 6744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:27.690467 6744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:27.690487 6744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:27.690500 6744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:27.696256 6744 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:37267
I20260928 04:43:27.696334 6871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:37267 every 8 connection(s)
I20260928 04:43:27.696676 6744 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
I20260928 04:43:27.699484 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 6744
I20260928 04:43:27.699605 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance
I20260928 04:43:27.699810 1931 external_mini_cluster.cc:1428] Setting key e9611cf2428d4f25e8d644e655b092e8
I20260928 04:43:27.700018 6596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.697441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:40733 (local address 127.1.226.254:42177)
0928 04:43:27.697510 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:27.697512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:27.697775 (+ 263us) server_negotiation.cc:407] Connection header received
0928 04:43:27.698246 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:27.698247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:27.698253 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:27.698318 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:27.699646 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:27.699648 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:27.699648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:43:27.699654 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:27.699680 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:43:27.699684 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:27.699739 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:27.699920 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 04:43:27.699935 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 04:43:27.700883 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:0
--local_ip_for_outbound_sockets=127.1.226.195
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:42177
--builtin_ntp_servers=127.1.226.212:34055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 04:43:27.703933 6874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.697435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42177 (local address 127.1.226.194:40733)
0928 04:43:27.697630 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:27.697664 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:27.698158 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:27.698388 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:27.698392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:27.698400 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:27.699442 (+ 1042us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:43:27.699443 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:27.699486 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:43:27.699522 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:27.699742 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:27.699747 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:27.699750 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:43:27.699839 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 04:43:27.703741 (+ 3902us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":98,"threads_started":1}
I20260928 04:43:27.704602 6872 heartbeater.cc:347] Connected to a master server at 127.1.226.254:42177
I20260928 04:43:27.704703 6872 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:27.704907 6872 heartbeater.cc:511] Master 127.1.226.254:42177 requested a full tablet report, sending...
I20260928 04:43:27.705327 6552 ts_manager.cc:194] Registered new tserver with Master: 369f570c04e44b8282d1cc652031e93b (127.1.226.194:37267)
W20260928 04:43:27.788792 6875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:27.789081 6875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:27.789124 6875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:27.790518 6875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:27.790591 6875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:27.790647 6875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:43:27.792246 6875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:34055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:42177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:27.792557 6875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:27.792891 6875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:27.798525 6883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:27.798520 6881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:27.798525 6880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:27.798810 6875 server_base.cc:1034] running on GCE node
I20260928 04:43:27.799038 6875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:27.799322 6875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:27.800462 6875 hybrid_clock.cc:648] HybridClock initialized: now 1790570607800448 us; error 33 us; skew 500 ppm
I20260928 04:43:27.801698 6875 webserver.cc:460] Webserver started at http://127.1.226.195:37793/ using document root <none> and password file <none>
I20260928 04:43:27.802023 6875 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:27.802073 6875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:27.802205 6875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:27.803396 6875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-2/data/instance:
uuid: "f8371f72eeff48fa8be471b7383bd672"
format_stamp: "Formatted at 2026-09-28 04:43:27 on dist-test-slave-j5c3"
server_key: "d7d946e439f2e8d2f34984ab5b71b1f8"
server_key_iv: "44b314625a944791c8924caf77ef7250"
server_key_version: "encryptionkey@0"
I20260928 04:43:27.803756 6875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance:
uuid: "f8371f72eeff48fa8be471b7383bd672"
format_stamp: "Formatted at 2026-09-28 04:43:27 on dist-test-slave-j5c3"
server_key: "d7d946e439f2e8d2f34984ab5b71b1f8"
server_key_iv: "44b314625a944791c8924caf77ef7250"
server_key_version: "encryptionkey@0"
I20260928 04:43:27.805227 6875 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:43:27.806027 6889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:27.806212 6875 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:27.806283 6875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "f8371f72eeff48fa8be471b7383bd672"
format_stamp: "Formatted at 2026-09-28 04:43:27 on dist-test-slave-j5c3"
server_key: "d7d946e439f2e8d2f34984ab5b71b1f8"
server_key_iv: "44b314625a944791c8924caf77ef7250"
server_key_version: "encryptionkey@0"
I20260928 04:43:27.806377 6875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:27.810452 6875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:27.810762 6875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:27.810899 6875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:27.811115 6875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:27.811496 6875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:27.811529 6875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:27.811551 6875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:27.811596 6875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:27.818179 6875 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:35735
I20260928 04:43:27.818218 7002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:35735 every 8 connection(s)
I20260928 04:43:27.818575 6875 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
I20260928 04:43:27.821895 6596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.819351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:58001 (local address 127.1.226.254:42177)
0928 04:43:27.819417 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:27.819419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:27.819646 (+ 227us) server_negotiation.cc:407] Connection header received
0928 04:43:27.820262 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:27.820263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:27.820272 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:27.820346 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:27.821603 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:27.821605 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:27.821605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:43:27.821611 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:27.821633 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:43:27.821637 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:27.821706 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:27.821802 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 04:43:27.821820 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 04:43:27.822118 7005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.819300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42177 (local address 127.1.226.195:58001)
0928 04:43:27.819517 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:27.819554 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:27.820104 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:27.820427 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:27.820432 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:27.820447 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:27.821429 (+ 982us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:43:27.821429 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:27.821461 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:43:27.821492 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:27.821701 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:27.821706 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:27.821711 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:43:27.821816 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 04:43:27.821885 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":73,"threads_started":1}
I20260928 04:43:27.822913 7003 heartbeater.cc:347] Connected to a master server at 127.1.226.254:42177
I20260928 04:43:27.823058 7003 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:27.823244 7003 heartbeater.cc:511] Master 127.1.226.254:42177 requested a full tablet report, sending...
I20260928 04:43:27.823670 6552 ts_manager.cc:194] Registered new tserver with Master: f8371f72eeff48fa8be471b7383bd672 (127.1.226.195:35735)
I20260928 04:43:27.826294 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 6875
I20260928 04:43:27.826391 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance
I20260928 04:43:27.826529 1931 external_mini_cluster.cc:1428] Setting key fdf36cce13d8c2f8d963ae81715b9bd2
I20260928 04:43:27.827509 1931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:43:27.830155 6596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.829119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41250 (local address 127.1.226.254:42177)
0928 04:43:27.829176 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:27.829178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:27.829301 (+ 123us) server_negotiation.cc:407] Connection header received
0928 04:43:27.829372 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:27.829373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:27.829378 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:27.829420 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:27.829980 (+ 560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:27.829981 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:27.829981 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:43:27.829984 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:27.830000 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:43:27.830004 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:27.830045 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:27.830082 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 04:43:27.830096 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 04:43:27.831817 6552 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:43:27.833537 6552 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:43:27.837638 7011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.835923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:43405 (local address 127.0.0.1:49870)
0928 04:43:27.836259 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:27.836304 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:27.836392 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:27.836582 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:27.836589 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:27.836597 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:27.837202 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:43:27.837203 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:27.837217 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:43:27.837237 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:27.837481 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:27.837487 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:27.837490 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:43:27.837550 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 04:43:27.837564 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":60,"threads_started":1}
I20260928 04:43:27.837658 7012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.836029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49870 (local address 127.1.226.193:43405)
0928 04:43:27.836240 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:27.836245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:27.836372 (+ 127us) server_negotiation.cc:407] Connection header received
0928 04:43:27.836478 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:27.836483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:27.836493 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:27.836574 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:27.837344 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:27.837353 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:27.837353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:43:27.837358 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:27.837392 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:43:27.837416 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:27.837484 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:27.837568 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 04:43:27.837582 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":112,"threads_started":1}
I20260928 04:43:27.838131 7014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.836313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:35735 (local address 127.0.0.1:44678)
0928 04:43:27.836505 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:27.836514 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:27.836547 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:27.837111 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:27.837111 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:27.837114 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:27.837748 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:43:27.837749 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:27.837754 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:43:27.837768 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:27.838010 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:27.838011 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:27.838012 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 04:43:27.838058 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 04:43:27.838070 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":81,"threads_started":1}
I20260928 04:43:27.838203 7015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.836407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44678 (local address 127.1.226.195:35735)
0928 04:43:27.836989 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:27.836994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:27.837004 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:43:27.837029 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:27.837031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:27.837039 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:27.837110 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:27.837866 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:27.837874 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:27.837874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:43:27.837878 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:27.837919 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:43:27.837940 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:27.837988 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:27.838085 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 04:43:27.838103 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":77,"threads_started":1}
I20260928 04:43:27.838619 7013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.836109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:37267 (local address 127.0.0.1:49376)
0928 04:43:27.836311 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:27.836320 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:27.836369 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:27.837586 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:27.837587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:27.837590 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:27.838234 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:43:27.838235 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:27.838241 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:43:27.838255 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:27.838486 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:27.838487 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:27.838487 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 04:43:27.838547 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 04:43:27.838560 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":106,"threads_started":1}
I20260928 04:43:27.838630 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.836471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49376 (local address 127.1.226.194:37267)
0928 04:43:27.837410 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:27.837415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:27.837425 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:43:27.837451 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:27.837453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:27.837462 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:27.837565 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:27.838348 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:27.838356 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:27.838356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:43:27.838360 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:27.838397 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:43:27.838416 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:27.838462 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:27.838559 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 04:43:27.838571 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":897,"thread_start_us":79,"threads_started":1}
I20260928 04:43:27.838703 6675 tablet_service.cc:1467] Processing CreateTablet for tablet 1f435fb8b0cb40b2bea54beb05fbae86 (DEFAULT_TABLE table=test-table [id=203e08bd229740539ef5fd594ce59a49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:27.839150 6675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f435fb8b0cb40b2bea54beb05fbae86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:27.841127 6937 tablet_service.cc:1467] Processing CreateTablet for tablet 1f435fb8b0cb40b2bea54beb05fbae86 (DEFAULT_TABLE table=test-table [id=203e08bd229740539ef5fd594ce59a49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:27.841554 6937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f435fb8b0cb40b2bea54beb05fbae86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:27.842811 7017 tablet_bootstrap.cc:492] T 1f435fb8b0cb40b2bea54beb05fbae86 P 6830682b21ed40b6a9f0cb21d7f2e94c: Bootstrap starting.
I20260928 04:43:27.843464 7017 tablet_bootstrap.cc:654] T 1f435fb8b0cb40b2bea54beb05fbae86 P 6830682b21ed40b6a9f0cb21d7f2e94c: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:27.843772 7017 log.cc:824] T 1f435fb8b0cb40b2bea54beb05fbae86 P 6830682b21ed40b6a9f0cb21d7f2e94c: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:27.844681 6806 tablet_service.cc:1467] Processing CreateTablet for tablet 1f435fb8b0cb40b2bea54beb05fbae86 (DEFAULT_TABLE table=test-table [id=203e08bd229740539ef5fd594ce59a49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:27.844939 7017 tablet_bootstrap.cc:492] T 1f435fb8b0cb40b2bea54beb05fbae86 P 6830682b21ed40b6a9f0cb21d7f2e94c: No bootstrap required, opened a new log
I20260928 04:43:27.845007 7017 ts_tablet_manager.cc:1397] T 1f435fb8b0cb40b2bea54beb05fbae86 P 6830682b21ed40b6a9f0cb21d7f2e94c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:43:27.845050 6806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f435fb8b0cb40b2bea54beb05fbae86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:27.845469 7019 tablet_bootstrap.cc:492] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672: Bootstrap starting.
I20260928 04:43:27.846033 7019 tablet_bootstrap.cc:654] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:27.846335 7019 log.cc:824] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:27.847067 7019 tablet_bootstrap.cc:492] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672: No bootstrap required, opened a new log
I20260928 04:43:27.847147 7019 ts_tablet_manager.cc:1397] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:43:27.847644 7017 raft_consensus.cc:348] T 1f435fb8b0cb40b2bea54beb05fbae86 P 6830682b21ed40b6a9f0cb21d7f2e94c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6830682b21ed40b6a9f0cb21d7f2e94c" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43405 } } peers { permanent_uuid: "369f570c04e44b8282d1cc652031e93b" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 37267 } } peers { permanent_uuid: "f8371f72eeff48fa8be471b7383bd672" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 35735 } }
I20260928 04:43:27.847854 7017 raft_consensus.cc:374] T 1f435fb8b0cb40b2bea54beb05fbae86 P 6830682b21ed40b6a9f0cb21d7f2e94c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:27.847894 7017 raft_consensus.cc:729] T 1f435fb8b0cb40b2bea54beb05fbae86 P 6830682b21ed40b6a9f0cb21d7f2e94c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6830682b21ed40b6a9f0cb21d7f2e94c, State: Initialized, Role: FOLLOWER
I20260928 04:43:27.848057 7017 consensus_queue.cc:260] T 1f435fb8b0cb40b2bea54beb05fbae86 P 6830682b21ed40b6a9f0cb21d7f2e94c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6830682b21ed40b6a9f0cb21d7f2e94c" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43405 } } peers { permanent_uuid: "369f570c04e44b8282d1cc652031e93b" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 37267 } } peers { permanent_uuid: "f8371f72eeff48fa8be471b7383bd672" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 35735 } }
I20260928 04:43:27.848388 7017 ts_tablet_manager.cc:1428] T 1f435fb8b0cb40b2bea54beb05fbae86 P 6830682b21ed40b6a9f0cb21d7f2e94c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 04:43:27.848678 6741 heartbeater.cc:503] Master 127.1.226.254:42177 was elected leader, sending a full tablet report...
I20260928 04:43:27.848785 7021 tablet_bootstrap.cc:492] T 1f435fb8b0cb40b2bea54beb05fbae86 P 369f570c04e44b8282d1cc652031e93b: Bootstrap starting.
I20260928 04:43:27.849501 7021 tablet_bootstrap.cc:654] T 1f435fb8b0cb40b2bea54beb05fbae86 P 369f570c04e44b8282d1cc652031e93b: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:27.849596 7019 raft_consensus.cc:348] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6830682b21ed40b6a9f0cb21d7f2e94c" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43405 } } peers { permanent_uuid: "369f570c04e44b8282d1cc652031e93b" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 37267 } } peers { permanent_uuid: "f8371f72eeff48fa8be471b7383bd672" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 35735 } }
I20260928 04:43:27.849802 7019 raft_consensus.cc:374] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:27.849829 7019 raft_consensus.cc:729] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8371f72eeff48fa8be471b7383bd672, State: Initialized, Role: FOLLOWER
I20260928 04:43:27.849846 7021 log.cc:824] T 1f435fb8b0cb40b2bea54beb05fbae86 P 369f570c04e44b8282d1cc652031e93b: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:27.849977 7019 consensus_queue.cc:260] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6830682b21ed40b6a9f0cb21d7f2e94c" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43405 } } peers { permanent_uuid: "369f570c04e44b8282d1cc652031e93b" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 37267 } } peers { permanent_uuid: "f8371f72eeff48fa8be471b7383bd672" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 35735 } }
I20260928 04:43:27.850276 7019 ts_tablet_manager.cc:1428] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:43:27.850296 7003 heartbeater.cc:503] Master 127.1.226.254:42177 was elected leader, sending a full tablet report...
I20260928 04:43:27.850673 7021 tablet_bootstrap.cc:492] T 1f435fb8b0cb40b2bea54beb05fbae86 P 369f570c04e44b8282d1cc652031e93b: No bootstrap required, opened a new log
I20260928 04:43:27.850751 7021 ts_tablet_manager.cc:1397] T 1f435fb8b0cb40b2bea54beb05fbae86 P 369f570c04e44b8282d1cc652031e93b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:43:27.852965 7021 raft_consensus.cc:348] T 1f435fb8b0cb40b2bea54beb05fbae86 P 369f570c04e44b8282d1cc652031e93b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6830682b21ed40b6a9f0cb21d7f2e94c" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43405 } } peers { permanent_uuid: "369f570c04e44b8282d1cc652031e93b" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 37267 } } peers { permanent_uuid: "f8371f72eeff48fa8be471b7383bd672" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 35735 } }
I20260928 04:43:27.853168 7021 raft_consensus.cc:374] T 1f435fb8b0cb40b2bea54beb05fbae86 P 369f570c04e44b8282d1cc652031e93b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:27.853206 7021 raft_consensus.cc:729] T 1f435fb8b0cb40b2bea54beb05fbae86 P 369f570c04e44b8282d1cc652031e93b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 369f570c04e44b8282d1cc652031e93b, State: Initialized, Role: FOLLOWER
I20260928 04:43:27.853343 7021 consensus_queue.cc:260] T 1f435fb8b0cb40b2bea54beb05fbae86 P 369f570c04e44b8282d1cc652031e93b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6830682b21ed40b6a9f0cb21d7f2e94c" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43405 } } peers { permanent_uuid: "369f570c04e44b8282d1cc652031e93b" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 37267 } } peers { permanent_uuid: "f8371f72eeff48fa8be471b7383bd672" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 35735 } }
I20260928 04:43:27.853609 7021 ts_tablet_manager.cc:1428] T 1f435fb8b0cb40b2bea54beb05fbae86 P 369f570c04e44b8282d1cc652031e93b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 04:43:27.853655 6872 heartbeater.cc:503] Master 127.1.226.254:42177 was elected leader, sending a full tablet report...
I20260928 04:43:27.913574 7024 raft_consensus.cc:482] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:43:27.913722 7024 raft_consensus.cc:504] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6830682b21ed40b6a9f0cb21d7f2e94c" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43405 } } peers { permanent_uuid: "369f570c04e44b8282d1cc652031e93b" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 37267 } } peers { permanent_uuid: "f8371f72eeff48fa8be471b7383bd672" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 35735 } }
I20260928 04:43:27.914181 7024 leader_election.cc:288] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6830682b21ed40b6a9f0cb21d7f2e94c (127.1.226.193:43405), 369f570c04e44b8282d1cc652031e93b (127.1.226.194:37267)
I20260928 04:43:27.915601 7005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.914455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:43405 (local address 127.1.226.195:41263)
0928 04:43:27.914540 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:27.914556 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:27.914613 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:27.914753 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:27.914754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:27.914756 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:27.915280 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:43:27.915280 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:27.915288 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:43:27.915301 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:27.915495 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:27.915496 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:27.915497 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 04:43:27.915539 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 04:43:27.915552 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 04:43:27.915868 7012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.914576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:41263 (local address 127.1.226.193:43405)
0928 04:43:27.914632 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:27.914634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:27.914643 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:43:27.914673 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:27.914673 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:27.914679 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:27.914735 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:27.915394 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:27.915395 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:27.915396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:43:27.915400 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:27.915420 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:43:27.915426 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:27.915492 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:27.915760 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 04:43:27.915780 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 04:43:27.915952 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.914594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:37267 (local address 127.1.226.195:53533)
0928 04:43:27.914798 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:27.914808 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:27.914888 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:27.915039 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:27.915040 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:27.915043 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:27.915473 (+ 430us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:43:27.915473 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:27.915480 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:43:27.915493 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:27.915785 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:27.915785 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:27.915786 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 04:43:27.915847 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 04:43:27.915859 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":115,"threads_started":1}
I20260928 04:43:27.915983 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:27.914743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:53533 (local address 127.1.226.194:37267)
0928 04:43:27.914792 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:27.914795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:27.914878 (+ 83us) server_negotiation.cc:407] Connection header received
0928 04:43:27.914951 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:27.914952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:27.914958 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:27.915035 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:27.915556 (+ 521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:27.915557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:27.915558 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:43:27.915562 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:27.915577 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:43:27.915581 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:27.915717 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:27.915867 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 04:43:27.915884 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 04:43:27.916225 6695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f435fb8b0cb40b2bea54beb05fbae86" candidate_uuid: "f8371f72eeff48fa8be471b7383bd672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6830682b21ed40b6a9f0cb21d7f2e94c" is_pre_election: true
I20260928 04:43:27.916290 6826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f435fb8b0cb40b2bea54beb05fbae86" candidate_uuid: "f8371f72eeff48fa8be471b7383bd672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "369f570c04e44b8282d1cc652031e93b" is_pre_election: true
I20260928 04:43:27.916388 6695 raft_consensus.cc:2454] T 1f435fb8b0cb40b2bea54beb05fbae86 P 6830682b21ed40b6a9f0cb21d7f2e94c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8371f72eeff48fa8be471b7383bd672 in term 0.
I20260928 04:43:27.916399 6826 raft_consensus.cc:2454] T 1f435fb8b0cb40b2bea54beb05fbae86 P 369f570c04e44b8282d1cc652031e93b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8371f72eeff48fa8be471b7383bd672 in term 0.
I20260928 04:43:27.916728 6890 leader_election.cc:302] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 369f570c04e44b8282d1cc652031e93b, f8371f72eeff48fa8be471b7383bd672; no voters:
I20260928 04:43:27.916865 7024 raft_consensus.cc:2781] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:43:27.916925 7024 raft_consensus.cc:482] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:43:27.916950 7024 raft_consensus.cc:3037] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672 [term 0 FOLLOWER]: Advancing to term 1
W20260928 04:43:27.947279 6873 tablet.cc:2369] T 1f435fb8b0cb40b2bea54beb05fbae86 P 369f570c04e44b8282d1cc652031e93b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:43:27.995800 7024 raft_consensus.cc:504] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6830682b21ed40b6a9f0cb21d7f2e94c" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43405 } } peers { permanent_uuid: "369f570c04e44b8282d1cc652031e93b" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 37267 } } peers { permanent_uuid: "f8371f72eeff48fa8be471b7383bd672" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 35735 } }
I20260928 04:43:27.996027 7024 leader_election.cc:288] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672 [CANDIDATE]: Term 1 election: Requested vote from peers 6830682b21ed40b6a9f0cb21d7f2e94c (127.1.226.193:43405), 369f570c04e44b8282d1cc652031e93b (127.1.226.194:37267)
I20260928 04:43:27.996307 6695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f435fb8b0cb40b2bea54beb05fbae86" candidate_uuid: "f8371f72eeff48fa8be471b7383bd672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6830682b21ed40b6a9f0cb21d7f2e94c"
I20260928 04:43:27.996340 6826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f435fb8b0cb40b2bea54beb05fbae86" candidate_uuid: "f8371f72eeff48fa8be471b7383bd672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "369f570c04e44b8282d1cc652031e93b"
I20260928 04:43:27.996450 6826 raft_consensus.cc:3037] T 1f435fb8b0cb40b2bea54beb05fbae86 P 369f570c04e44b8282d1cc652031e93b [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:27.996452 6695 raft_consensus.cc:3037] T 1f435fb8b0cb40b2bea54beb05fbae86 P 6830682b21ed40b6a9f0cb21d7f2e94c [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:27.997363 6695 raft_consensus.cc:2454] T 1f435fb8b0cb40b2bea54beb05fbae86 P 6830682b21ed40b6a9f0cb21d7f2e94c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8371f72eeff48fa8be471b7383bd672 in term 1.
I20260928 04:43:27.997573 6890 leader_election.cc:302] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6830682b21ed40b6a9f0cb21d7f2e94c, f8371f72eeff48fa8be471b7383bd672; no voters:
I20260928 04:43:27.997670 7024 raft_consensus.cc:2781] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:27.997663 6826 raft_consensus.cc:2454] T 1f435fb8b0cb40b2bea54beb05fbae86 P 369f570c04e44b8282d1cc652031e93b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8371f72eeff48fa8be471b7383bd672 in term 1.
I20260928 04:43:27.997855 7024 raft_consensus.cc:686] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672 [term 1 LEADER]: Becoming Leader. State: Replica: f8371f72eeff48fa8be471b7383bd672, State: Running, Role: LEADER
I20260928 04:43:27.997974 7024 consensus_queue.cc:237] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6830682b21ed40b6a9f0cb21d7f2e94c" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43405 } } peers { permanent_uuid: "369f570c04e44b8282d1cc652031e93b" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 37267 } } peers { permanent_uuid: "f8371f72eeff48fa8be471b7383bd672" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 35735 } }
I20260928 04:43:27.999081 6553 catalog_manager.cc:5495] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8371f72eeff48fa8be471b7383bd672 (127.1.226.195). New cstate: current_term: 1 leader_uuid: "f8371f72eeff48fa8be471b7383bd672" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6830682b21ed40b6a9f0cb21d7f2e94c" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "369f570c04e44b8282d1cc652031e93b" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8371f72eeff48fa8be471b7383bd672" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 35735 } health_report { overall_health: HEALTHY } } }
I20260928 04:43:28.022255 7015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:28.021224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44686 (local address 127.1.226.195:35735)
0928 04:43:28.021272 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:28.021274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:28.021283 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:43:28.021375 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:28.021375 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:28.021395 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:28.021446 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:28.022018 (+ 572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:28.022019 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:28.022020 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:43:28.022023 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:28.022038 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:43:28.022042 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:28.022089 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:28.022133 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 04:43:28.022145 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 04:43:28.024370 6826 raft_consensus.cc:1261] T 1f435fb8b0cb40b2bea54beb05fbae86 P 369f570c04e44b8282d1cc652031e93b [term 1 FOLLOWER]: Refusing update from remote peer f8371f72eeff48fa8be471b7383bd672: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:28.024371 6695 raft_consensus.cc:1261] T 1f435fb8b0cb40b2bea54beb05fbae86 P 6830682b21ed40b6a9f0cb21d7f2e94c [term 1 FOLLOWER]: Refusing update from remote peer f8371f72eeff48fa8be471b7383bd672: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:28.024802 7024 consensus_queue.cc:1035] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672 [LEADER]: Connected to new peer: Peer: permanent_uuid: "369f570c04e44b8282d1cc652031e93b" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:28.024973 7027 consensus_queue.cc:1035] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6830682b21ed40b6a9f0cb21d7f2e94c" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:28.028301 7031 mvcc.cc:205] Tried to move back new op lower bound from 7334177210463997952 to 7334177210360655872. Current Snapshot: MvccSnapshot[applied={T|T < 7334177210463997952}]
I20260928 04:43:28.029814 6553 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:43:28.029910 6553 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:43:28.030696 6553 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b15d7b7be37645538c152780f408435d: Sending DeleteTablet for 3 replicas of tablet 1f435fb8b0cb40b2bea54beb05fbae86
I20260928 04:43:28.031046 6675 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f435fb8b0cb40b2bea54beb05fbae86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:28 UTC) from {username='slave'} at 127.0.0.1:49870
I20260928 04:43:28.031070 6806 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f435fb8b0cb40b2bea54beb05fbae86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:28 UTC) from {username='slave'} at 127.0.0.1:49376
I20260928 04:43:28.031121 6937 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f435fb8b0cb40b2bea54beb05fbae86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:28 UTC) from {username='slave'} at 127.0.0.1:44678
I20260928 04:43:28.031311 7041 tablet_replica.cc:330] T 1f435fb8b0cb40b2bea54beb05fbae86 P 369f570c04e44b8282d1cc652031e93b: stopping tablet replica
I20260928 04:43:28.031419 7041 raft_consensus.cc:2229] T 1f435fb8b0cb40b2bea54beb05fbae86 P 369f570c04e44b8282d1cc652031e93b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:43:28.031533 7041 raft_consensus.cc:2258] T 1f435fb8b0cb40b2bea54beb05fbae86 P 369f570c04e44b8282d1cc652031e93b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:28.031623 7042 tablet_replica.cc:330] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672: stopping tablet replica
I20260928 04:43:28.031734 7042 raft_consensus.cc:2229] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:43:28.031828 7042 raft_consensus.cc:2258] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:28.031999 7041 ts_tablet_manager.cc:1905] T 1f435fb8b0cb40b2bea54beb05fbae86 P 369f570c04e44b8282d1cc652031e93b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:28.032250 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 6613
I20260928 04:43:28.032284 7042 ts_tablet_manager.cc:1905] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 04:43:28.038492 6537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.226.193:43405 (error 108)
I20260928 04:43:28.038825 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 6744
I20260928 04:43:28.039153 7014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:28.039001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:43405 (local address 127.0.0.1:49874)
0928 04:43:28.039057 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:28.039095 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.193:43405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 04:43:28.040751 6537 catalog_manager.cc:4584] TS 6830682b21ed40b6a9f0cb21d7f2e94c (127.1.226.193:43405): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1f435fb8b0cb40b2bea54beb05fbae86: Network error: Client connection negotiation failed: client connection to 127.1.226.193:43405: connect: Connection refused (error 111)
I20260928 04:43:28.093456 7014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:28.093293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:43405 (local address 127.0.0.1:49882)
0928 04:43:28.093369 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:28.093413 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.193:43405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 04:43:28.094038 7014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:28.093843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:43405 (local address 127.0.0.1:49886)
0928 04:43:28.093956 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:28.093996 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.193:43405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 04:43:28.097515 7042 ts_tablet_manager.cc:1918] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:43:28.097621 7042 log.cc:1196] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570585693787-1931-0/minicluster-data/ts-2/wal/wals/1f435fb8b0cb40b2bea54beb05fbae86
I20260928 04:43:28.097899 7042 ts_tablet_manager.cc:1939] T 1f435fb8b0cb40b2bea54beb05fbae86 P f8371f72eeff48fa8be471b7383bd672: Deleting consensus metadata
I20260928 04:43:28.100772 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 6875
I20260928 04:43:28.100905 7014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:28.100751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:37267 (local address 127.0.0.1:49378)
0928 04:43:28.100830 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:28.100865 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.194:37267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 04:43:28.126286 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 6522
I20260928 04:43:28.126374 7014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:28.126228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:35735 (local address 127.0.0.1:44700)
0928 04:43:28.126289 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:28.126321 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.195:35735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
2026-09-28T04:43:28Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (842 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 04:43:28.146323 1931 mini_oidc.cc:152] Starting JWKS server
I20260928 04:43:28.147544 1931 webserver.cc:460] Webserver started at https://127.0.0.1:39105/ using document root <none> and password file <none>
I20260928 04:43:28.147691 1931 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 04:43:28.147900 1931 webserver.cc:460] Webserver started at http://127.0.0.1:39435/ using document root <none> and password file <none>
2026-09-28T04:43:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:43:28Z Disabled control of system clock
I20260928 04:43:28.156340 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:33025
--webserver_interface=127.1.226.254
--webserver_port=0
--builtin_ntp_servers=127.1.226.212:43225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:33025
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39105/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570585693787-1931-0/testchainca.cert with env {}
W20260928 04:43:28.248082 7049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:28.248404 7049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:28.248430 7049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:28.249748 7049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:43:28.249794 7049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:28.249807 7049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:28.249820 7049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:43:28.249832 7049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 04:43:28.249845 7049 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 04:43:28.249900 7049 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39105/jwks.json
I20260928 04:43:28.251693 7049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:43225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:33025
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:33025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39105/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570585693787-1931-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:28.251981 7049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:28.252293 7049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:28.257670 7055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:28.257699 7057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:28.257670 7054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:28.257858 7049 server_base.cc:1034] running on GCE node
I20260928 04:43:28.258615 7049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:28.258963 7049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:28.260226 7049 hybrid_clock.cc:648] HybridClock initialized: now 1790570608260127 us; error 38 us; skew 500 ppm
I20260928 04:43:28.261731 7049 webserver.cc:460] Webserver started at http://127.1.226.254:45729/ using document root <none> and password file <none>
I20260928 04:43:28.262023 7049 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:28.262102 7049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:28.262267 7049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:28.263629 7049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "36f53e85479d4f4ea0f505f545f547f1"
format_stamp: "Formatted at 2026-09-28 04:43:28 on dist-test-slave-j5c3"
server_key: "2ab99cfa6cc80aced70a17d1374e7de2"
server_key_iv: "ef46450222e5946501fe8e2386396b2e"
server_key_version: "encryptionkey@0"
I20260928 04:43:28.264024 7049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "36f53e85479d4f4ea0f505f545f547f1"
format_stamp: "Formatted at 2026-09-28 04:43:28 on dist-test-slave-j5c3"
server_key: "2ab99cfa6cc80aced70a17d1374e7de2"
server_key_iv: "ef46450222e5946501fe8e2386396b2e"
server_key_version: "encryptionkey@0"
I20260928 04:43:28.265558 7049 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:43:28.266439 7063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:28.266615 7049 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:43:28.266734 7049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "36f53e85479d4f4ea0f505f545f547f1"
format_stamp: "Formatted at 2026-09-28 04:43:28 on dist-test-slave-j5c3"
server_key: "2ab99cfa6cc80aced70a17d1374e7de2"
server_key_iv: "ef46450222e5946501fe8e2386396b2e"
server_key_version: "encryptionkey@0"
I20260928 04:43:28.266824 7049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:28.303400 7049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:28.303804 7049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:28.303953 7049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:28.308899 7049 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:33025
I20260928 04:43:28.308988 7115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:33025 every 8 connection(s)
I20260928 04:43:28.309353 7049 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:43:28.310396 7116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:28.311820 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 7049
I20260928 04:43:28.311942 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:43:28.312117 1931 external_mini_cluster.cc:1428] Setting key 0093b6d046e220e4fd203dfb1d6457c8
I20260928 04:43:28.313352 7116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36f53e85479d4f4ea0f505f545f547f1: Bootstrap starting.
I20260928 04:43:28.314085 7116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36f53e85479d4f4ea0f505f545f547f1: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:28.314384 7116 log.cc:824] T 00000000000000000000000000000000 P 36f53e85479d4f4ea0f505f545f547f1: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:28.315164 7116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36f53e85479d4f4ea0f505f545f547f1: No bootstrap required, opened a new log
I20260928 04:43:28.317548 7116 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36f53e85479d4f4ea0f505f545f547f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36f53e85479d4f4ea0f505f545f547f1" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33025 } }
I20260928 04:43:28.317734 7116 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36f53e85479d4f4ea0f505f545f547f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:28.317766 7116 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36f53e85479d4f4ea0f505f545f547f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36f53e85479d4f4ea0f505f545f547f1, State: Initialized, Role: FOLLOWER
I20260928 04:43:28.317914 7116 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36f53e85479d4f4ea0f505f545f547f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36f53e85479d4f4ea0f505f545f547f1" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33025 } }
I20260928 04:43:28.317991 7116 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36f53e85479d4f4ea0f505f545f547f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:43:28.318041 7116 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36f53e85479d4f4ea0f505f545f547f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:43:28.318089 7116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36f53e85479d4f4ea0f505f545f547f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:28.318550 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:28.312709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49824 (local address 127.1.226.254:33025)
0928 04:43:28.312926 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:28.312935 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:28.312961 (+ 26us) server_negotiation.cc:407] Connection header received
0928 04:43:28.313415 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:28.313423 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:28.313542 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:28.313674 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:28.313911 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:28.315148 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:28.316262 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:28.316752 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:28.316862 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:28.317801 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:28.317806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:28.317807 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:43:28.318048 (+ 241us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:28.318096 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:43:28.318112 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:28.318224 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:28.318269 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 04:43:28.318361 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260928 04:43:28.319527 1931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:43:28.404258 7116 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36f53e85479d4f4ea0f505f545f547f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36f53e85479d4f4ea0f505f545f547f1" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33025 } }
I20260928 04:43:28.404454 7116 leader_election.cc:302] T 00000000000000000000000000000000 P 36f53e85479d4f4ea0f505f545f547f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36f53e85479d4f4ea0f505f545f547f1; no voters:
I20260928 04:43:28.404783 7116 leader_election.cc:288] T 00000000000000000000000000000000 P 36f53e85479d4f4ea0f505f545f547f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:43:28.404846 7121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36f53e85479d4f4ea0f505f545f547f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:28.404978 7121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36f53e85479d4f4ea0f505f545f547f1 [term 1 LEADER]: Becoming Leader. State: Replica: 36f53e85479d4f4ea0f505f545f547f1, State: Running, Role: LEADER
I20260928 04:43:28.405072 7121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36f53e85479d4f4ea0f505f545f547f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36f53e85479d4f4ea0f505f545f547f1" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33025 } }
I20260928 04:43:28.405129 7116 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36f53e85479d4f4ea0f505f545f547f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:43:28.405440 7121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36f53e85479d4f4ea0f505f545f547f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36f53e85479d4f4ea0f505f545f547f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36f53e85479d4f4ea0f505f545f547f1" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33025 } } }
I20260928 04:43:28.405489 7122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36f53e85479d4f4ea0f505f545f547f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36f53e85479d4f4ea0f505f545f547f1. Latest consensus state: current_term: 1 leader_uuid: "36f53e85479d4f4ea0f505f545f547f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36f53e85479d4f4ea0f505f545f547f1" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33025 } } }
I20260928 04:43:28.405877 7122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36f53e85479d4f4ea0f505f545f547f1 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:28.405526 7121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36f53e85479d4f4ea0f505f545f547f1 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:28.406437 7130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:43:28.407001 7130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:43:28.409546 7130 catalog_manager.cc:1347] Generated new cluster ID: 1893c51b6e3240efae61eaa6b28156c0
I20260928 04:43:28.409617 7130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:43:28.420830 7130 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:43:28.421612 7130 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:43:28.427174 7130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36f53e85479d4f4ea0f505f545f547f1: Generated new TSK 0
I20260928 04:43:28.427390 7130 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 04:43:28.459391 7146 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 04:43:28.459573 7146 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 04:43:28.459789 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:28.436890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49836 (local address 127.1.226.254:33025)
0928 04:43:28.436947 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:28.436949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:28.437095 (+ 146us) server_negotiation.cc:407] Connection header received
0928 04:43:28.437246 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:28.437247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:28.437259 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:28.437345 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0928 04:43:28.437580 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:28.438192 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:28.439498 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:28.439764 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:28.439833 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:28.440105 (+ 272us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 04:43:28.459374 (+ 19269us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39105/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 04:43:28.459700 (+ 326us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49836: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39105/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 04:43:28.459856 7119 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49836: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39105/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 04:43:28.460047 7145 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:28.436777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:33025 (local address 127.0.0.1:49836)
0928 04:43:28.437003 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:28.437016 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:28.437108 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:28.437380 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:28.437381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:28.437384 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0928 04:43:28.437472 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:28.437474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:28.438315 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:28.438316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:28.439392 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:28.439395 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:28.439865 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:28.439866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:28.439940 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:28.439965 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 04:43:28.459537 (+ 19572us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39105/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 04:43:28.459684 (+ 147us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.226.254:33025: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39105/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":94,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.226.254:33025: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39105/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 04:43:28.460649 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 7049
2026-09-28T04:43:28Z chronyd exiting
I20260928 04:43:28.497301 1931 test_util.cc:175] -----------------------------------------------
I20260928 04:43:28.497394 1931 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570585693787-1931-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (352 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 04:43:28.498162 1931 mini_oidc.cc:152] Starting JWKS server
I20260928 04:43:28.498826 1931 webserver.cc:460] Webserver started at https://127.0.0.1:45667/ using document root <none> and password file <none>
I20260928 04:43:28.498868 1931 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 04:43:28.499013 1931 webserver.cc:460] Webserver started at http://127.0.0.1:38781/ using document root <none> and password file <none>
2026-09-28T04:43:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:43:28Z Disabled control of system clock
I20260928 04:43:28.507130 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:40869
--webserver_interface=127.1.226.254
--webserver_port=0
--builtin_ntp_servers=127.1.226.212:40849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:40869
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45667/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570585693787-1931-0/testchainca.cert with env {}
W20260928 04:43:28.590590 7152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:28.590811 7152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:28.590834 7152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:28.592200 7152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:43:28.592259 7152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:28.592273 7152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:28.592286 7152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:43:28.592299 7152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 04:43:28.592330 7152 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 04:43:28.592356 7152 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45667/jwks.json
I20260928 04:43:28.593922 7152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:40849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:40869
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:40869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45667/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570585693787-1931-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:28.594210 7152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:28.594437 7152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:28.599483 7158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:28.599517 7157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:28.599766 7152 server_base.cc:1034] running on GCE node
W20260928 04:43:28.599473 7160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:28.600077 7152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:28.600427 7152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:28.601583 7152 hybrid_clock.cc:648] HybridClock initialized: now 1790570608601579 us; error 38 us; skew 500 ppm
I20260928 04:43:28.602916 7152 webserver.cc:460] Webserver started at http://127.1.226.254:45509/ using document root <none> and password file <none>
I20260928 04:43:28.603178 7152 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:28.603250 7152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:28.603412 7152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:28.604620 7152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "f9105ddf6784409191f27965ad3bc107"
format_stamp: "Formatted at 2026-09-28 04:43:28 on dist-test-slave-j5c3"
server_key: "d6b6057532ce15b2e2b3d6de8f2096c0"
server_key_iv: "bd40cfc201341e60360490bc1cad442b"
server_key_version: "encryptionkey@0"
I20260928 04:43:28.604996 7152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "f9105ddf6784409191f27965ad3bc107"
format_stamp: "Formatted at 2026-09-28 04:43:28 on dist-test-slave-j5c3"
server_key: "d6b6057532ce15b2e2b3d6de8f2096c0"
server_key_iv: "bd40cfc201341e60360490bc1cad442b"
server_key_version: "encryptionkey@0"
I20260928 04:43:28.606328 7152 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:43:28.607097 7166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:28.607295 7152 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:28.607402 7152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "f9105ddf6784409191f27965ad3bc107"
format_stamp: "Formatted at 2026-09-28 04:43:28 on dist-test-slave-j5c3"
server_key: "d6b6057532ce15b2e2b3d6de8f2096c0"
server_key_iv: "bd40cfc201341e60360490bc1cad442b"
server_key_version: "encryptionkey@0"
I20260928 04:43:28.607483 7152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:28.621873 7152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:28.622259 7152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:28.622424 7152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:28.626513 7152 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:40869
I20260928 04:43:28.626544 7218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:40869 every 8 connection(s)
I20260928 04:43:28.626935 7152 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:43:28.627955 7219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:28.630362 7219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9105ddf6784409191f27965ad3bc107: Bootstrap starting.
I20260928 04:43:28.630985 7219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9105ddf6784409191f27965ad3bc107: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:28.631274 7219 log.cc:824] T 00000000000000000000000000000000 P f9105ddf6784409191f27965ad3bc107: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:28.632066 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 7152
I20260928 04:43:28.632202 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:43:28.632197 7219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9105ddf6784409191f27965ad3bc107: No bootstrap required, opened a new log
I20260928 04:43:28.632398 1931 external_mini_cluster.cc:1428] Setting key fc9c2f5f18e43f98c899fcf4a50abcea
I20260928 04:43:28.634127 7219 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9105ddf6784409191f27965ad3bc107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9105ddf6784409191f27965ad3bc107" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 40869 } }
I20260928 04:43:28.634275 7219 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9105ddf6784409191f27965ad3bc107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:28.634299 7219 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9105ddf6784409191f27965ad3bc107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9105ddf6784409191f27965ad3bc107, State: Initialized, Role: FOLLOWER
I20260928 04:43:28.634456 7219 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9105ddf6784409191f27965ad3bc107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9105ddf6784409191f27965ad3bc107" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 40869 } }
I20260928 04:43:28.634554 7219 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9105ddf6784409191f27965ad3bc107 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:43:28.634588 7219 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9105ddf6784409191f27965ad3bc107 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:43:28.634648 7219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9105ddf6784409191f27965ad3bc107 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:28.638867 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:28.632992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54984 (local address 127.1.226.254:40869)
0928 04:43:28.633227 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:28.633234 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:28.633259 (+ 25us) server_negotiation.cc:407] Connection header received
0928 04:43:28.634117 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:28.634128 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:28.634230 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:28.634424 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:28.634701 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:28.635971 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:28.636927 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:28.637375 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:28.637479 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:28.638210 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:28.638214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:28.638214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:43:28.638431 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:28.638465 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:43:28.638477 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:28.638533 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:28.638617 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 04:43:28.638690 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":93,"threads_started":1}
I20260928 04:43:28.639937 1931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:43:28.708120 7219 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9105ddf6784409191f27965ad3bc107 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9105ddf6784409191f27965ad3bc107" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 40869 } }
I20260928 04:43:28.708376 7219 leader_election.cc:302] T 00000000000000000000000000000000 P f9105ddf6784409191f27965ad3bc107 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9105ddf6784409191f27965ad3bc107; no voters:
I20260928 04:43:28.708631 7219 leader_election.cc:288] T 00000000000000000000000000000000 P f9105ddf6784409191f27965ad3bc107 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:43:28.708781 7224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9105ddf6784409191f27965ad3bc107 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:28.708972 7224 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9105ddf6784409191f27965ad3bc107 [term 1 LEADER]: Becoming Leader. State: Replica: f9105ddf6784409191f27965ad3bc107, State: Running, Role: LEADER
I20260928 04:43:28.709002 7219 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9105ddf6784409191f27965ad3bc107 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:43:28.709146 7224 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9105ddf6784409191f27965ad3bc107 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9105ddf6784409191f27965ad3bc107" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 40869 } }
I20260928 04:43:28.709628 7225 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9105ddf6784409191f27965ad3bc107 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9105ddf6784409191f27965ad3bc107" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9105ddf6784409191f27965ad3bc107" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 40869 } } }
I20260928 04:43:28.709887 7225 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9105ddf6784409191f27965ad3bc107 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:28.709813 7226 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9105ddf6784409191f27965ad3bc107 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9105ddf6784409191f27965ad3bc107. Latest consensus state: current_term: 1 leader_uuid: "f9105ddf6784409191f27965ad3bc107" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9105ddf6784409191f27965ad3bc107" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 40869 } } }
I20260928 04:43:28.710143 7226 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9105ddf6784409191f27965ad3bc107 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:28.710319 7233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:43:28.710908 7233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:43:28.713238 7233 catalog_manager.cc:1347] Generated new cluster ID: 67f059750826400f92960ab3f251b0ca
I20260928 04:43:28.713330 7233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:43:28.730293 7233 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:43:28.731096 7233 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:43:28.735026 7233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9105ddf6784409191f27965ad3bc107: Generated new TSK 0
I20260928 04:43:28.735288 7233 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 04:43:28.762807 7249 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 04:43:28.762910 7249 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 04:43:28.763258 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:28.748287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54990 (local address 127.1.226.254:40869)
0928 04:43:28.748355 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:28.748358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:28.748463 (+ 105us) server_negotiation.cc:407] Connection header received
0928 04:43:28.748538 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:28.748539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:28.748563 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:28.748659 (+ 96us) server_negotiation.cc:226] Negotiated authn=JWT
0928 04:43:28.748861 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:28.749666 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:28.750455 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:28.750779 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:28.750850 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:28.751019 (+ 169us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 04:43:28.763015 (+ 11996us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45667/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 04:43:28.763181 (+ 166us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54990: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45667/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 04:43:28.763339 7223 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54990: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45667/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 04:43:28.763377 7248 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:28.748224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:40869 (local address 127.0.0.1:54990)
0928 04:43:28.748376 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:28.748388 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:28.748471 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:28.748682 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:28.748683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:28.748685 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 04:43:28.748783 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:28.748785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:28.749768 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:28.749769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:28.750369 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:28.750371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:28.750859 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:28.750860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:28.750905 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:28.750910 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 04:43:28.763161 (+ 12251us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45667/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 04:43:28.763272 (+ 111us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.226.254:40869: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45667/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
I20260928 04:43:28.763784 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 7152
2026-09-28T04:43:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (304 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 04:43:28.802702 1931 mini_oidc.cc:152] Starting JWKS server
I20260928 04:43:28.803401 1931 webserver.cc:460] Webserver started at https://127.0.0.1:46759/ using document root <none> and password file <none>
I20260928 04:43:28.803478 1931 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 04:43:28.803658 1931 webserver.cc:460] Webserver started at http://127.0.0.1:33969/ using document root <none> and password file <none>
2026-09-28T04:43:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:43:28Z Disabled control of system clock
I20260928 04:43:28.811753 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:45447
--webserver_interface=127.1.226.254
--webserver_port=0
--builtin_ntp_servers=127.1.226.212:35061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:45447
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46759/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 04:43:28.898070 7255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:28.898334 7255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:28.898370 7255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:28.899644 7255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:43:28.899719 7255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:28.899740 7255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:28.899761 7255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:43:28.899791 7255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 04:43:28.899821 7255 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 04:43:28.899844 7255 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46759/jwks.json
I20260928 04:43:28.901830 7255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:35061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:45447
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:45447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46759/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:28.902134 7255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:28.902491 7255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:28.907634 7261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:28.907697 7263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:28.907644 7260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:28.907837 7255 server_base.cc:1034] running on GCE node
I20260928 04:43:28.908277 7255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:28.908610 7255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:28.909806 7255 hybrid_clock.cc:648] HybridClock initialized: now 1790570608909787 us; error 42 us; skew 500 ppm
I20260928 04:43:28.911372 7255 webserver.cc:460] Webserver started at http://127.1.226.254:39979/ using document root <none> and password file <none>
I20260928 04:43:28.911640 7255 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:28.911705 7255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:28.911842 7255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:28.913403 7255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "0c68f184edb749808202c501a992d81e"
format_stamp: "Formatted at 2026-09-28 04:43:28 on dist-test-slave-j5c3"
server_key: "cde353b119f46f1f2225d35a5d545971"
server_key_iv: "50ea59496cee10fad81531db0e9d2491"
server_key_version: "encryptionkey@0"
I20260928 04:43:28.913826 7255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "0c68f184edb749808202c501a992d81e"
format_stamp: "Formatted at 2026-09-28 04:43:28 on dist-test-slave-j5c3"
server_key: "cde353b119f46f1f2225d35a5d545971"
server_key_iv: "50ea59496cee10fad81531db0e9d2491"
server_key_version: "encryptionkey@0"
I20260928 04:43:28.915220 7255 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:43:28.915997 7269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:28.916249 7255 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:28.916371 7255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "0c68f184edb749808202c501a992d81e"
format_stamp: "Formatted at 2026-09-28 04:43:28 on dist-test-slave-j5c3"
server_key: "cde353b119f46f1f2225d35a5d545971"
server_key_iv: "50ea59496cee10fad81531db0e9d2491"
server_key_version: "encryptionkey@0"
I20260928 04:43:28.916462 7255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:28.925628 7255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:28.925971 7255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:28.926128 7255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:28.930183 7255 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:45447
I20260928 04:43:28.930218 7321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:45447 every 8 connection(s)
I20260928 04:43:28.930605 7255 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:43:28.931344 7322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:28.934808 7322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c68f184edb749808202c501a992d81e: Bootstrap starting.
I20260928 04:43:28.935449 7322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c68f184edb749808202c501a992d81e: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:28.935751 7322 log.cc:824] T 00000000000000000000000000000000 P 0c68f184edb749808202c501a992d81e: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:28.936594 7322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c68f184edb749808202c501a992d81e: No bootstrap required, opened a new log
I20260928 04:43:28.936672 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 7255
I20260928 04:43:28.936865 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:43:28.937058 1931 external_mini_cluster.cc:1428] Setting key e7c9799b33de4535080ff970777e735b
I20260928 04:43:28.938709 7322 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c68f184edb749808202c501a992d81e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c68f184edb749808202c501a992d81e" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45447 } }
I20260928 04:43:28.938889 7322 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c68f184edb749808202c501a992d81e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:28.938933 7322 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c68f184edb749808202c501a992d81e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c68f184edb749808202c501a992d81e, State: Initialized, Role: FOLLOWER
I20260928 04:43:28.939067 7322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c68f184edb749808202c501a992d81e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c68f184edb749808202c501a992d81e" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45447 } }
I20260928 04:43:28.939167 7322 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c68f184edb749808202c501a992d81e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:43:28.939213 7322 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c68f184edb749808202c501a992d81e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:43:28.939273 7322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c68f184edb749808202c501a992d81e [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:28.943361 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:28.937643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48084 (local address 127.1.226.254:45447)
0928 04:43:28.937863 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:28.937871 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:28.937896 (+ 25us) server_negotiation.cc:407] Connection header received
0928 04:43:28.938700 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:28.938709 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:28.938782 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:28.938948 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:28.939174 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:28.940547 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:28.941367 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:28.941809 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:28.941959 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:28.942634 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:28.942638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:28.942638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:43:28.942853 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:28.942891 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:43:28.942904 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:28.942988 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:28.943067 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 04:43:28.943160 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
I20260928 04:43:28.944442 1931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:43:29.012737 7322 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c68f184edb749808202c501a992d81e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c68f184edb749808202c501a992d81e" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45447 } }
I20260928 04:43:29.012981 7322 leader_election.cc:302] T 00000000000000000000000000000000 P 0c68f184edb749808202c501a992d81e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c68f184edb749808202c501a992d81e; no voters:
I20260928 04:43:29.013244 7322 leader_election.cc:288] T 00000000000000000000000000000000 P 0c68f184edb749808202c501a992d81e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:43:29.013339 7327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c68f184edb749808202c501a992d81e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:29.013566 7327 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c68f184edb749808202c501a992d81e [term 1 LEADER]: Becoming Leader. State: Replica: 0c68f184edb749808202c501a992d81e, State: Running, Role: LEADER
I20260928 04:43:29.013646 7322 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c68f184edb749808202c501a992d81e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:43:29.013726 7327 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c68f184edb749808202c501a992d81e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c68f184edb749808202c501a992d81e" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45447 } }
I20260928 04:43:29.014256 7328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c68f184edb749808202c501a992d81e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c68f184edb749808202c501a992d81e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c68f184edb749808202c501a992d81e" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45447 } } }
I20260928 04:43:29.014405 7328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c68f184edb749808202c501a992d81e [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:29.014647 7329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c68f184edb749808202c501a992d81e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c68f184edb749808202c501a992d81e. Latest consensus state: current_term: 1 leader_uuid: "0c68f184edb749808202c501a992d81e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c68f184edb749808202c501a992d81e" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45447 } } }
I20260928 04:43:29.014799 7329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c68f184edb749808202c501a992d81e [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:29.015275 7336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:43:29.015877 7336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:43:29.017634 7336 catalog_manager.cc:1347] Generated new cluster ID: 7aeb002ad9fd4acc97e45478d092b699
I20260928 04:43:29.017714 7336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:43:29.032184 7336 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:43:29.032948 7336 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:43:29.039876 7336 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c68f184edb749808202c501a992d81e: Generated new TSK 0
I20260928 04:43:29.040143 7336 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:43:29.057578 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:29.053551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48086 (local address 127.1.226.254:45447)
0928 04:43:29.053689 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:29.053691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:29.053776 (+ 85us) server_negotiation.cc:407] Connection header received
0928 04:43:29.053833 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:29.053834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:29.053848 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:29.053925 (+ 77us) server_negotiation.cc:226] Negotiated authn=JWT
0928 04:43:29.054079 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:29.054601 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:29.055831 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:29.056021 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:29.056083 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:29.056317 (+ 234us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 04:43:29.057342 (+ 1025us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46759/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 04:43:29.057486 (+ 144us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48086: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46759/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 04:43:29.057643 7326 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48086: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46759/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 04:43:29.057674 7351 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:29.053466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:45447 (local address 127.0.0.1:48086)
0928 04:43:29.053665 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:29.053678 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:29.053764 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:29.053942 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:29.053943 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:29.053946 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0928 04:43:29.054009 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:29.054011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:29.054754 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:29.054755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:29.055723 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:29.055726 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:29.056091 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:29.056093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:29.056190 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:29.056196 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 04:43:29.057449 (+ 1253us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46759/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 04:43:29.057568 (+ 119us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.226.254:45447: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46759/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":114,"threads_started":1}
I20260928 04:43:29.057920 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 7255
E20260928 04:43:29.070125 7352 webserver.cc:533] Webserver: sslize failed:
E20260928 04:43:29.070240 7352 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T04:43:29Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (294 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T04:43:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:43:29Z Disabled control of system clock
I20260928 04:43:29.104449 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:36093
--webserver_interface=127.1.226.254
--webserver_port=0
--builtin_ntp_servers=127.1.226.212:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:36093
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570585693787-1931-0/testchainca.cert with env {}
W20260928 04:43:29.191443 7356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:29.191725 7356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:29.191753 7356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:29.193132 7356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:43:29.193192 7356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:29.193207 7356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:29.193221 7356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:43:29.193235 7356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 04:43:29.193249 7356 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 04:43:29.193261 7356 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 04:43:29.195032 7356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:36093
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:36093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570585693787-1931-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:29.195257 7356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:29.195438 7356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:29.200878 7361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:29.200939 7362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:29.201110 7364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:29.201045 7356 server_base.cc:1034] running on GCE node
I20260928 04:43:29.201395 7356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:29.201751 7356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:29.202917 7356 hybrid_clock.cc:648] HybridClock initialized: now 1790570609202898 us; error 49 us; skew 500 ppm
I20260928 04:43:29.204444 7356 webserver.cc:460] Webserver started at http://127.1.226.254:43503/ using document root <none> and password file <none>
I20260928 04:43:29.204730 7356 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:29.204803 7356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:29.204973 7356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:29.206182 7356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "9dbc01912d1d4bb69517fd4baf27b3fb"
format_stamp: "Formatted at 2026-09-28 04:43:29 on dist-test-slave-j5c3"
server_key: "e0a1bba3ab2de276cb626d9e4481059c"
server_key_iv: "4a233303020545c61c90e737e698c567"
server_key_version: "encryptionkey@0"
I20260928 04:43:29.206584 7356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "9dbc01912d1d4bb69517fd4baf27b3fb"
format_stamp: "Formatted at 2026-09-28 04:43:29 on dist-test-slave-j5c3"
server_key: "e0a1bba3ab2de276cb626d9e4481059c"
server_key_iv: "4a233303020545c61c90e737e698c567"
server_key_version: "encryptionkey@0"
I20260928 04:43:29.208088 7356 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:43:29.209080 7370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:29.209302 7356 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:29.209432 7356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "9dbc01912d1d4bb69517fd4baf27b3fb"
format_stamp: "Formatted at 2026-09-28 04:43:29 on dist-test-slave-j5c3"
server_key: "e0a1bba3ab2de276cb626d9e4481059c"
server_key_iv: "4a233303020545c61c90e737e698c567"
server_key_version: "encryptionkey@0"
I20260928 04:43:29.209532 7356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:29.244143 7356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:29.244573 7356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:29.244732 7356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:29.248879 7356 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:36093
I20260928 04:43:29.248941 7422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:36093 every 8 connection(s)
I20260928 04:43:29.249323 7356 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:43:29.250362 7423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:29.252965 7423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dbc01912d1d4bb69517fd4baf27b3fb: Bootstrap starting.
I20260928 04:43:29.253592 7423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dbc01912d1d4bb69517fd4baf27b3fb: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:29.253887 7423 log.cc:824] T 00000000000000000000000000000000 P 9dbc01912d1d4bb69517fd4baf27b3fb: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:29.254630 7423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dbc01912d1d4bb69517fd4baf27b3fb: No bootstrap required, opened a new log
I20260928 04:43:29.256438 7423 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dbc01912d1d4bb69517fd4baf27b3fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbc01912d1d4bb69517fd4baf27b3fb" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 36093 } }
I20260928 04:43:29.256588 7423 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dbc01912d1d4bb69517fd4baf27b3fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:29.256615 7423 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dbc01912d1d4bb69517fd4baf27b3fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dbc01912d1d4bb69517fd4baf27b3fb, State: Initialized, Role: FOLLOWER
I20260928 04:43:29.256794 7423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dbc01912d1d4bb69517fd4baf27b3fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbc01912d1d4bb69517fd4baf27b3fb" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 36093 } }
I20260928 04:43:29.256856 7423 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dbc01912d1d4bb69517fd4baf27b3fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:43:29.256911 7423 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dbc01912d1d4bb69517fd4baf27b3fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:43:29.256964 7423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dbc01912d1d4bb69517fd4baf27b3fb [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:29.259447 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 7356
I20260928 04:43:29.259562 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:43:29.259786 1931 external_mini_cluster.cc:1428] Setting key ca8b91898107c85ce14847b46eab2fb6
I20260928 04:43:29.265143 7427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:29.260391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49100 (local address 127.1.226.254:36093)
0928 04:43:29.260690 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:29.260698 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:29.260729 (+ 31us) server_negotiation.cc:407] Connection header received
0928 04:43:29.260846 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:29.260852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:29.260912 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:29.261042 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:29.261209 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:29.262117 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:29.262976 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:29.263487 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:29.263605 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:29.264369 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:29.264373 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:29.264374 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:43:29.264583 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:29.264623 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:43:29.264636 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:29.264696 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:29.264762 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 04:43:29.264838 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":99,"threads_started":1}
I20260928 04:43:29.266567 1931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:43:29.313041 7423 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dbc01912d1d4bb69517fd4baf27b3fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbc01912d1d4bb69517fd4baf27b3fb" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 36093 } }
I20260928 04:43:29.313256 7423 leader_election.cc:302] T 00000000000000000000000000000000 P 9dbc01912d1d4bb69517fd4baf27b3fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dbc01912d1d4bb69517fd4baf27b3fb; no voters:
I20260928 04:43:29.313539 7423 leader_election.cc:288] T 00000000000000000000000000000000 P 9dbc01912d1d4bb69517fd4baf27b3fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:43:29.313637 7428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dbc01912d1d4bb69517fd4baf27b3fb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:29.313817 7428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dbc01912d1d4bb69517fd4baf27b3fb [term 1 LEADER]: Becoming Leader. State: Replica: 9dbc01912d1d4bb69517fd4baf27b3fb, State: Running, Role: LEADER
I20260928 04:43:29.313907 7423 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dbc01912d1d4bb69517fd4baf27b3fb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:43:29.313927 7428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dbc01912d1d4bb69517fd4baf27b3fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbc01912d1d4bb69517fd4baf27b3fb" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 36093 } }
I20260928 04:43:29.314519 7429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbc01912d1d4bb69517fd4baf27b3fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dbc01912d1d4bb69517fd4baf27b3fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbc01912d1d4bb69517fd4baf27b3fb" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 36093 } } }
I20260928 04:43:29.314702 7429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbc01912d1d4bb69517fd4baf27b3fb [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:29.314513 7430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbc01912d1d4bb69517fd4baf27b3fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dbc01912d1d4bb69517fd4baf27b3fb. Latest consensus state: current_term: 1 leader_uuid: "9dbc01912d1d4bb69517fd4baf27b3fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbc01912d1d4bb69517fd4baf27b3fb" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 36093 } } }
I20260928 04:43:29.314889 7430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbc01912d1d4bb69517fd4baf27b3fb [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:29.315150 7437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:43:29.315752 7437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:43:29.317766 7437 catalog_manager.cc:1347] Generated new cluster ID: 384f4748891b4cfa9194dbd30db17378
I20260928 04:43:29.317839 7437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:43:29.373461 7437 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:43:29.374142 7437 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:43:29.386256 7437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dbc01912d1d4bb69517fd4baf27b3fb: Generated new TSK 0
I20260928 04:43:29.386511 7437 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:43:29.843824 7427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:29.408811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49106 (local address 127.1.226.254:36093)
0928 04:43:29.408869 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:29.408871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:29.408992 (+ 121us) server_negotiation.cc:407] Connection header received
0928 04:43:29.409111 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:29.409112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:29.409122 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:29.409194 (+ 72us) server_negotiation.cc:226] Negotiated authn=JWT
0928 04:43:29.409347 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:29.409843 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:29.410728 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:29.410932 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:29.411009 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:29.411197 (+ 188us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 04:43:29.843573 (+432376us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 04:43:29.843749 (+ 176us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49106: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 04:43:29.843889 7427 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49106: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 04:43:29.844004 7452 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:29.408741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36093 (local address 127.0.0.1:49106)
0928 04:43:29.408921 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:29.408931 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:29.409012 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:29.409199 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:29.409200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:29.409201 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0928 04:43:29.409264 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:29.409266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:29.409954 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:29.409955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:29.410567 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:29.410569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:29.411014 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:29.411014 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:29.411071 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:29.411077 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 04:43:29.843724 (+432647us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 04:43:29.843853 (+ 129us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.226.254:36093: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":90,"threads_started":1}
I20260928 04:43:29.844359 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 7356
2026-09-28T04:43:29Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (765 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (92 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (96 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:43:32 dist-test-slave-j5c3 krb5kdc[7464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:43:32 dist-test-slave-j5c3 krb5kdc[7464](info): set up 2 sockets
Sep 28 04:43:32 dist-test-slave-j5c3 krb5kdc[7464](info): commencing operation
krb5kdc: starting...
W20260928 04:43:33.052551 1931 mini_kdc.cc:121] Time spent starting KDC: real 3.000s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:43:33 dist-test-slave-j5c3 krb5kdc[7464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:43:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:43:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:43:33.176426 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:35683
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:39875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:35683
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:33.263916 7477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:33.264127 7477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:33.264166 7477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:33.265383 7477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:43:33.265432 7477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:33.265447 7477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:33.265460 7477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:43:33.265471 7477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:43:33.266987 7477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:39875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:35683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:35683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:33.267254 7477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:33.267493 7477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:33.272447 7483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:33.272471 7482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:33.272488 7477 server_base.cc:1034] running on GCE node
W20260928 04:43:33.272471 7485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:33.272957 7477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:33.273298 7477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:33.274466 7477 hybrid_clock.cc:648] HybridClock initialized: now 1790570613274463 us; error 40 us; skew 500 ppm
Sep 28 04:43:33 dist-test-slave-j5c3 krb5kdc[7464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570613, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:33.282742 7477 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:43:33.283206 7477 webserver.cc:460] Webserver started at http://127.1.226.254:41405/ using document root <none> and password file <none>
I20260928 04:43:33.283457 7477 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:33.283536 7477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:33.283689 7477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:33.284832 7477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "de16ef1c1c574e529f73df32a1a8a0c7"
format_stamp: "Formatted at 2026-09-28 04:43:33 on dist-test-slave-j5c3"
server_key: "20f2331902c2600d3645abbbbf94e847"
server_key_iv: "a53500bdbb869832b66c7d70106dd4ba"
server_key_version: "encryptionkey@0"
I20260928 04:43:33.285203 7477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "de16ef1c1c574e529f73df32a1a8a0c7"
format_stamp: "Formatted at 2026-09-28 04:43:33 on dist-test-slave-j5c3"
server_key: "20f2331902c2600d3645abbbbf94e847"
server_key_iv: "a53500bdbb869832b66c7d70106dd4ba"
server_key_version: "encryptionkey@0"
I20260928 04:43:33.286541 7477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:43:33.287307 7492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:33.287508 7477 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:43:33.287580 7477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "de16ef1c1c574e529f73df32a1a8a0c7"
format_stamp: "Formatted at 2026-09-28 04:43:33 on dist-test-slave-j5c3"
server_key: "20f2331902c2600d3645abbbbf94e847"
server_key_iv: "a53500bdbb869832b66c7d70106dd4ba"
server_key_version: "encryptionkey@0"
I20260928 04:43:33.287667 7477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:33.298122 7477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:33.300015 7477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:33.300200 7477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:33.303962 7477 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:35683
I20260928 04:43:33.304059 7544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:35683 every 8 connection(s)
I20260928 04:43:33.304437 7477 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:43:33.305325 7545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:33.307606 7545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de16ef1c1c574e529f73df32a1a8a0c7: Bootstrap starting.
I20260928 04:43:33.308142 7545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de16ef1c1c574e529f73df32a1a8a0c7: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:33.308480 7545 log.cc:824] T 00000000000000000000000000000000 P de16ef1c1c574e529f73df32a1a8a0c7: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:33.309208 7545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de16ef1c1c574e529f73df32a1a8a0c7: No bootstrap required, opened a new log
I20260928 04:43:33.310823 7545 raft_consensus.cc:348] T 00000000000000000000000000000000 P de16ef1c1c574e529f73df32a1a8a0c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de16ef1c1c574e529f73df32a1a8a0c7" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 35683 } }
I20260928 04:43:33.310959 7545 raft_consensus.cc:374] T 00000000000000000000000000000000 P de16ef1c1c574e529f73df32a1a8a0c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:33.310983 7545 raft_consensus.cc:729] T 00000000000000000000000000000000 P de16ef1c1c574e529f73df32a1a8a0c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de16ef1c1c574e529f73df32a1a8a0c7, State: Initialized, Role: FOLLOWER
I20260928 04:43:33.311147 7545 consensus_queue.cc:260] T 00000000000000000000000000000000 P de16ef1c1c574e529f73df32a1a8a0c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de16ef1c1c574e529f73df32a1a8a0c7" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 35683 } }
I20260928 04:43:33.311231 7545 raft_consensus.cc:388] T 00000000000000000000000000000000 P de16ef1c1c574e529f73df32a1a8a0c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:43:33.311254 7545 raft_consensus.cc:482] T 00000000000000000000000000000000 P de16ef1c1c574e529f73df32a1a8a0c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:43:33.311316 7545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de16ef1c1c574e529f73df32a1a8a0c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:33.311573 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 7477
I20260928 04:43:33.311702 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:43:33.311898 1931 external_mini_cluster.cc:1428] Setting key 0ad8193328e84a271c6f819195bec26d
I20260928 04:43:33.311935 7545 raft_consensus.cc:504] T 00000000000000000000000000000000 P de16ef1c1c574e529f73df32a1a8a0c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de16ef1c1c574e529f73df32a1a8a0c7" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 35683 } }
I20260928 04:43:33.312057 7545 leader_election.cc:302] T 00000000000000000000000000000000 P de16ef1c1c574e529f73df32a1a8a0c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de16ef1c1c574e529f73df32a1a8a0c7; no voters:
I20260928 04:43:33.312259 7545 leader_election.cc:288] T 00000000000000000000000000000000 P de16ef1c1c574e529f73df32a1a8a0c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:43:33.312290 7548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de16ef1c1c574e529f73df32a1a8a0c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:33.312505 7548 raft_consensus.cc:686] T 00000000000000000000000000000000 P de16ef1c1c574e529f73df32a1a8a0c7 [term 1 LEADER]: Becoming Leader. State: Replica: de16ef1c1c574e529f73df32a1a8a0c7, State: Running, Role: LEADER
I20260928 04:43:33.312577 7545 sys_catalog.cc:564] T 00000000000000000000000000000000 P de16ef1c1c574e529f73df32a1a8a0c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:43:33.312691 7548 consensus_queue.cc:237] T 00000000000000000000000000000000 P de16ef1c1c574e529f73df32a1a8a0c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de16ef1c1c574e529f73df32a1a8a0c7" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 35683 } }
I20260928 04:43:33.313110 7551 sys_catalog.cc:455] T 00000000000000000000000000000000 P de16ef1c1c574e529f73df32a1a8a0c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de16ef1c1c574e529f73df32a1a8a0c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de16ef1c1c574e529f73df32a1a8a0c7" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 35683 } } }
I20260928 04:43:33.313397 7551 sys_catalog.cc:458] T 00000000000000000000000000000000 P de16ef1c1c574e529f73df32a1a8a0c7 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:33.313102 7548 sys_catalog.cc:455] T 00000000000000000000000000000000 P de16ef1c1c574e529f73df32a1a8a0c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de16ef1c1c574e529f73df32a1a8a0c7. Latest consensus state: current_term: 1 leader_uuid: "de16ef1c1c574e529f73df32a1a8a0c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de16ef1c1c574e529f73df32a1a8a0c7" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 35683 } } }
I20260928 04:43:33.313689 7548 sys_catalog.cc:458] T 00000000000000000000000000000000 P de16ef1c1c574e529f73df32a1a8a0c7 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:33.313880 7559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:43:33.314458 7559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:43:33.316195 7559 catalog_manager.cc:1347] Generated new cluster ID: 8985996250f44defadac246d92788046
I20260928 04:43:33.316241 7559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:43:33 dist-test-slave-j5c3 krb5kdc[7464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:33.322842 7552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:33.312835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44686 (local address 127.1.226.254:35683)
0928 04:43:33.312996 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:33.313008 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:33.313347 (+ 339us) server_negotiation.cc:407] Connection header received
0928 04:43:33.313586 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:33.313596 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:33.313654 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:33.313773 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:33.314616 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:33.315625 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:33.316620 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:33.317094 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:33.317228 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:33.319510 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:33.319514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:33.319514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:33.319526 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:33.321246 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:33.321635 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:33.321638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:33.321639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:33.321733 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:33.321947 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:33.321948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:33.321948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:33.322204 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:33.322390 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:33.322469 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 04:43:33.322631 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":71,"threads_started":1}
I20260928 04:43:33.325328 7559 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:43:33.325892 7559 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:43:33.329252 7559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de16ef1c1c574e529f73df32a1a8a0c7: Generated new TSK 0
I20260928 04:43:33.329478 7559 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.226.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
I20260928 04:43:33.359139 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:0
--local_ip_for_outbound_sockets=127.1.226.193
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:35683
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:39875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:33.455547 7571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:33.455754 7571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:33.455778 7571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:33.457017 7571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:33.457062 7571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:33.457089 7571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:43:33.458565 7571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:39875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:35683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:33.458803 7571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:33.459052 7571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:33.464793 7577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:33.465004 7576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:33.464823 7579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:33.465152 7571 server_base.cc:1034] running on GCE node
I20260928 04:43:33.465425 7571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:33.465703 7571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:33.466871 7571 hybrid_clock.cc:648] HybridClock initialized: now 1790570613466845 us; error 42 us; skew 500 ppm
Sep 28 04:43:33 dist-test-slave-j5c3 krb5kdc[7464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570613, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:33.470008 7571 init.cc:377] Logged in from keytab as kudu/127.1.226.193@KRBTEST.COM (short username kudu)
I20260928 04:43:33.470508 7571 webserver.cc:460] Webserver started at http://127.1.226.193:42663/ using document root <none> and password file <none>
I20260928 04:43:33.470744 7571 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:33.470820 7571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:33.470973 7571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:33.472043 7571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-0/data/instance:
uuid: "94df63572cc041cc8ff651239a35b7f2"
format_stamp: "Formatted at 2026-09-28 04:43:33 on dist-test-slave-j5c3"
server_key: "7f1d4028f15a606d5eeaa8be8fff8902"
server_key_iv: "dfdcf98a3f84a067706693b7542858cc"
server_key_version: "encryptionkey@0"
I20260928 04:43:33.472493 7571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance:
uuid: "94df63572cc041cc8ff651239a35b7f2"
format_stamp: "Formatted at 2026-09-28 04:43:33 on dist-test-slave-j5c3"
server_key: "7f1d4028f15a606d5eeaa8be8fff8902"
server_key_iv: "dfdcf98a3f84a067706693b7542858cc"
server_key_version: "encryptionkey@0"
I20260928 04:43:33.473853 7571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:33.474607 7586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:33.474802 7571 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:33.474910 7571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "94df63572cc041cc8ff651239a35b7f2"
format_stamp: "Formatted at 2026-09-28 04:43:33 on dist-test-slave-j5c3"
server_key: "7f1d4028f15a606d5eeaa8be8fff8902"
server_key_iv: "dfdcf98a3f84a067706693b7542858cc"
server_key_version: "encryptionkey@0"
I20260928 04:43:33.474982 7571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:33.484258 7571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:33.486152 7571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:33.486325 7571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:33.486562 7571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:33.486882 7571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:33.486936 7571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:33.486971 7571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:33.487017 7571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:33.493337 7571 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:34231
I20260928 04:43:33.493384 7699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:34231 every 8 connection(s)
I20260928 04:43:33.493717 7571 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
I20260928 04:43:33.494758 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 7571
I20260928 04:43:33.494900 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance
I20260928 04:43:33.495100 1931 external_mini_cluster.cc:1428] Setting key 55376a02db704a4774c08294a5d5a328
Sep 28 04:43:33 dist-test-slave-j5c3 krb5kdc[7464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570613, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.226.194@KRBTEST.COM; defaulting to no policy
I20260928 04:43:33.505201 7552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:33.494592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:58145 (local address 127.1.226.254:35683)
0928 04:43:33.494709 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:33.494712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:33.494882 (+ 170us) server_negotiation.cc:407] Connection header received
0928 04:43:33.496038 (+ 1156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:33.496039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:33.496054 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:33.496141 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:33.496881 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:33.497676 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:33.499205 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:33.499408 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:33.499473 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:33.502181 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:33.502183 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:33.502183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:33.502192 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:33.503666 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:33.504196 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:33.504197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:33.504198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:33.504253 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:33.504512 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:33.504513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:33.504514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:33.504635 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:33.504709 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:33.505011 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 04:43:33.505097 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 04:43:33.505388 7702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:33.494475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:35683 (local address 127.1.226.193:58145)
0928 04:43:33.494715 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:33.494766 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:33.495901 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:33.496242 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:33.496247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:33.496547 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:33.496757 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:33.496762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:33.497802 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:33.497803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:33.499065 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:33.499068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:33.499506 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:33.499507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:33.499683 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:33.500372 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:33.500383 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:33.502059 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:33.503771 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:33.503778 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:33.503781 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:33.504074 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:33.504322 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:33.504322 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:33.504323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:33.504442 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:33.504712 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:33.504719 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:33.504866 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 04:43:33.504977 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 04:43:33.505143 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":122,"threads_started":1}
I20260928 04:43:33.505908 7700 heartbeater.cc:347] Connected to a master server at 127.1.226.254:35683
I20260928 04:43:33.506019 7700 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:33.506202 7700 heartbeater.cc:511] Master 127.1.226.254:35683 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.194@KRBTEST.COM" created.
I20260928 04:43:33.506835 7509 ts_manager.cc:194] Registered new tserver with Master: 94df63572cc041cc8ff651239a35b7f2 (127.1.226.193:34231)
I20260928 04:43:33.507738 7509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.193@KRBTEST.COM'} at 127.1.226.193:58145
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
I20260928 04:43:33.523072 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:0
--local_ip_for_outbound_sockets=127.1.226.194
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:35683
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:39875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:33.604671 7705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:33.604882 7705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:33.604905 7705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:33.606016 7705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:33.606052 7705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:33.606078 7705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:43:33.607560 7705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:39875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:35683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:33.607751 7705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:33.607944 7705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:33.612995 7710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:33.613000 7713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:33.613004 7711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:33.613271 7705 server_base.cc:1034] running on GCE node
I20260928 04:43:33.613575 7705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:33.613772 7705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:33.614914 7705 hybrid_clock.cc:648] HybridClock initialized: now 1790570613614900 us; error 32 us; skew 500 ppm
Sep 28 04:43:33 dist-test-slave-j5c3 krb5kdc[7464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570613, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:33.617393 7705 init.cc:377] Logged in from keytab as kudu/127.1.226.194@KRBTEST.COM (short username kudu)
I20260928 04:43:33.617875 7705 webserver.cc:460] Webserver started at http://127.1.226.194:42695/ using document root <none> and password file <none>
I20260928 04:43:33.618060 7705 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:33.618099 7705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:33.618184 7705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:33.619210 7705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-1/data/instance:
uuid: "85b3c1af0a8b44d98ece1a161c87d305"
format_stamp: "Formatted at 2026-09-28 04:43:33 on dist-test-slave-j5c3"
server_key: "12cde8004336a84a42d8d20e28833488"
server_key_iv: "71315c10da26daf0038b55dd68c505e2"
server_key_version: "encryptionkey@0"
I20260928 04:43:33.619509 7705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance:
uuid: "85b3c1af0a8b44d98ece1a161c87d305"
format_stamp: "Formatted at 2026-09-28 04:43:33 on dist-test-slave-j5c3"
server_key: "12cde8004336a84a42d8d20e28833488"
server_key_iv: "71315c10da26daf0038b55dd68c505e2"
server_key_version: "encryptionkey@0"
I20260928 04:43:33.620841 7705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:43:33.621538 7720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:33.621752 7705 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:43:33.621870 7705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "85b3c1af0a8b44d98ece1a161c87d305"
format_stamp: "Formatted at 2026-09-28 04:43:33 on dist-test-slave-j5c3"
server_key: "12cde8004336a84a42d8d20e28833488"
server_key_iv: "71315c10da26daf0038b55dd68c505e2"
server_key_version: "encryptionkey@0"
I20260928 04:43:33.621950 7705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:33.633798 7705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:33.635738 7705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:33.635914 7705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:33.636178 7705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:33.636521 7705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:33.636554 7705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:33.636576 7705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:33.636620 7705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:33.642472 7705 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:34895
I20260928 04:43:33.642540 7833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:34895 every 8 connection(s)
I20260928 04:43:33.642891 7705 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
I20260928 04:43:33.648039 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 7705
I20260928 04:43:33.648134 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance
I20260928 04:43:33.648353 1931 external_mini_cluster.cc:1428] Setting key 38e7c22a691c826068f2f82402a91ea2
Sep 28 04:43:33 dist-test-slave-j5c3 krb5kdc[7464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570613, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:33.653591 7552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:33.643695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:53977 (local address 127.1.226.254:35683)
0928 04:43:33.643791 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:33.643793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:33.644003 (+ 210us) server_negotiation.cc:407] Connection header received
0928 04:43:33.645123 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:33.645124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:33.645138 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:33.645218 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:33.646001 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:33.646565 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:33.647682 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:33.647902 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:33.647998 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:33.650512 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:33.650514 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:33.650514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:33.650521 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:33.651926 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:33.652518 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:33.652519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:33.652520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:33.652577 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:33.652905 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:33.652906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:33.652907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:33.653025 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:33.653079 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:33.653451 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 04:43:33.653524 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 04:43:33.653904 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:33.643667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:35683 (local address 127.1.226.194:53977)
0928 04:43:33.643848 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:33.643898 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:33.644979 (+ 1081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:33.645341 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:33.645345 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:33.645629 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:33.645888 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:33.645893 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:33.646670 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:33.646672 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:33.647577 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:33.647580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:33.648006 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:33.648007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:33.648215 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:33.648821 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:33.648831 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:33.650418 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:33.652013 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:33.652021 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:33.652023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:33.652425 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:33.652656 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:33.652658 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:33.652658 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:33.652801 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:33.653095 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:33.653101 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:33.653296 (+ 195us) client_negotiation.cc:769] Sending connection context
0928 04:43:33.653417 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 04:43:33.653610 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":76,"threads_started":1}
I20260928 04:43:33.654425 7834 heartbeater.cc:347] Connected to a master server at 127.1.226.254:35683
I20260928 04:43:33.654536 7834 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:33.654738 7834 heartbeater.cc:511] Master 127.1.226.254:35683 requested a full tablet report, sending...
I20260928 04:43:33.655148 7509 ts_manager.cc:194] Registered new tserver with Master: 85b3c1af0a8b44d98ece1a161c87d305 (127.1.226.194:34895)
I20260928 04:43:33.655539 7509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.194@KRBTEST.COM'} at 127.1.226.194:53977
WARNING: no policy specified for kudu/127.1.226.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
I20260928 04:43:33.674719 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:0
--local_ip_for_outbound_sockets=127.1.226.195
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:35683
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:39875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:33.755905 7839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:33.756160 7839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:33.756223 7839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:33.757478 7839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:33.757560 7839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:33.757608 7839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:43:33.759204 7839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:39875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:35683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:33.759495 7839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:33.759748 7839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:33.764932 7845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:33.765003 7839 server_base.cc:1034] running on GCE node
W20260928 04:43:33.764998 7844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:33.764914 7847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:33.765364 7839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:33.765636 7839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:33.766798 7839 hybrid_clock.cc:648] HybridClock initialized: now 1790570613766777 us; error 33 us; skew 500 ppm
Sep 28 04:43:33 dist-test-slave-j5c3 krb5kdc[7464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570613, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:33.769665 7839 init.cc:377] Logged in from keytab as kudu/127.1.226.195@KRBTEST.COM (short username kudu)
I20260928 04:43:33.770184 7839 webserver.cc:460] Webserver started at http://127.1.226.195:36307/ using document root <none> and password file <none>
I20260928 04:43:33.770440 7839 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:33.770532 7839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:33.770685 7839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:33.771785 7839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-2/data/instance:
uuid: "636a002c08b44d80a99e9482bdd83f44"
format_stamp: "Formatted at 2026-09-28 04:43:33 on dist-test-slave-j5c3"
server_key: "d7868f5b4d6ed8cc2ab697ae9067bee5"
server_key_iv: "9974f198e56b08b9ae1fb53253ed4f65"
server_key_version: "encryptionkey@0"
I20260928 04:43:33.772239 7839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance:
uuid: "636a002c08b44d80a99e9482bdd83f44"
format_stamp: "Formatted at 2026-09-28 04:43:33 on dist-test-slave-j5c3"
server_key: "d7868f5b4d6ed8cc2ab697ae9067bee5"
server_key_iv: "9974f198e56b08b9ae1fb53253ed4f65"
server_key_version: "encryptionkey@0"
I20260928 04:43:33.773666 7839 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:33.774775 7854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:33.775035 7839 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:33.775189 7839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "636a002c08b44d80a99e9482bdd83f44"
format_stamp: "Formatted at 2026-09-28 04:43:33 on dist-test-slave-j5c3"
server_key: "d7868f5b4d6ed8cc2ab697ae9067bee5"
server_key_iv: "9974f198e56b08b9ae1fb53253ed4f65"
server_key_version: "encryptionkey@0"
I20260928 04:43:33.775280 7839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:33.792950 7839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:33.795017 7839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:33.795212 7839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:33.795461 7839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:33.795794 7839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:33.795848 7839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:33.795904 7839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:33.795941 7839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:33.802268 7839 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:46819
I20260928 04:43:33.802309 7967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:46819 every 8 connection(s)
I20260928 04:43:33.802650 7839 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
Sep 28 04:43:33 dist-test-slave-j5c3 krb5kdc[7464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570613, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:33.810009 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 7839
I20260928 04:43:33.810114 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance
I20260928 04:43:33.810307 1931 external_mini_cluster.cc:1428] Setting key fdaca5716744f2e6009cbd84ba4d94cf
I20260928 04:43:33.813378 7552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:33.803435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:48377 (local address 127.1.226.254:35683)
0928 04:43:33.803479 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:33.803484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:33.803724 (+ 240us) server_negotiation.cc:407] Connection header received
0928 04:43:33.804864 (+ 1140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:33.804866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:33.804880 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:33.804938 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:33.805622 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:33.806122 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:33.807167 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:33.807485 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:33.807739 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:33.810105 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:33.810107 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:33.810107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:33.810149 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:33.811842 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:33.812304 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:33.812305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:33.812305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:33.812372 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:33.812655 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:33.812656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:33.812656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:33.812788 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:33.812883 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:33.813163 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 04:43:33.813249 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:43:33.813511 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:33.803417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:35683 (local address 127.1.226.195:48377)
0928 04:43:33.803586 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:33.803622 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:33.804671 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:33.805022 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:33.805025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:33.805288 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:33.805518 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:33.805522 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:33.806235 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:33.806236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:33.807069 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:33.807071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:33.807755 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:33.807756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:33.807899 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:33.808513 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:33.808522 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:33.810000 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:33.811929 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:33.811936 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:33.811938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:33.812232 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:33.812438 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:33.812439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:33.812439 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:33.812561 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:33.812855 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:33.812861 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:33.813014 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 04:43:33.813118 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 04:43:33.813278 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260928 04:43:33.813983 7968 heartbeater.cc:347] Connected to a master server at 127.1.226.254:35683
I20260928 04:43:33.814086 7968 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:33.814266 7968 heartbeater.cc:511] Master 127.1.226.254:35683 requested a full tablet report, sending...
I20260928 04:43:33.814643 7509 ts_manager.cc:194] Registered new tserver with Master: 636a002c08b44d80a99e9482bdd83f44 (127.1.226.195:46819)
I20260928 04:43:33.815112 7509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.195@KRBTEST.COM'} at 127.1.226.195:48377
I20260928 04:43:33.822439 1931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/test-admin.keytab.
Sep 28 04:43:33 dist-test-slave-j5c3 krb5kdc[7464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:33.838668 1931 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 04:43:33.840673 7975 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:33.839757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48156 (local address 127.1.226.193:34231)
0928 04:43:33.839991 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:33.839998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:33.840008 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:43:33.840046 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:33.840048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:33.840062 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:33.840153 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:33.840582 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48156: BlockingRecv error: recv got EOF from 127.0.0.1:48156 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":92,"threads_started":1}
W20260928 04:43:33.840863 1931 init.cc:244] Error while opening credential cache '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/krb5cc)
Sep 28 04:43:33 dist-test-slave-j5c3 krb5kdc[7464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:33.841930 1931 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 04:43:33 dist-test-slave-j5c3 krb5kdc[7464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:33.843063 1931 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 04:43:33 dist-test-slave-j5c3 krb5kdc[7464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.193@KRBTEST.COM
I20260928 04:43:33.852435 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:33.843780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48168 (local address 127.1.226.193:34231)
0928 04:43:33.843865 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:33.843867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:33.843883 (+ 16us) server_negotiation.cc:407] Connection header received
0928 04:43:33.844001 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:33.844002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:33.844018 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:33.844091 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:33.844658 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:33.845492 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:33.846411 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:33.846814 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:33.846910 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:33.849396 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:33.849410 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:33.849410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:33.849425 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:33.851065 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:33.851528 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:33.851532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:33.851533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:33.851589 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:33.851805 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:33.851806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:33.851806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:33.851967 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:33.852136 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:33.852222 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 04:43:33.852311 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 04:43:33.854406 7975 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:33.853871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48180 (local address 127.1.226.193:34231)
0928 04:43:33.853938 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:33.853939 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:33.853960 (+ 21us) server_negotiation.cc:407] Connection header received
0928 04:43:33.854013 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:33.854014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:33.854024 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:33.854063 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:33.854371 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48180: BlockingRecv error: recv got EOF from 127.0.0.1:48180 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 04:43:33.854488 1931 init.cc:244] Error while opening credential cache '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/krb5cc)
Sep 28 04:43:33 dist-test-slave-j5c3 krb5kdc[7464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:33.855584 1931 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 04:43:33 dist-test-slave-j5c3 krb5kdc[7464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:33.856534 1931 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 04:43:33 dist-test-slave-j5c3 krb5kdc[7464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.193@KRBTEST.COM
I20260928 04:43:33.863468 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:33.857100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48188 (local address 127.1.226.193:34231)
0928 04:43:33.857135 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:33.857137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:33.857191 (+ 54us) server_negotiation.cc:407] Connection header received
0928 04:43:33.857268 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:33.857269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:33.857278 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:33.857332 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:33.857673 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:33.858155 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:33.858845 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:33.859029 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:33.859078 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:33.861091 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:33.861092 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:33.861093 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:33.861099 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:33.862457 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:33.862783 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:33.862783 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:33.862784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:33.862833 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:33.862994 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:33.862994 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:33.862994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:33.863079 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:33.863146 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:33.863245 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 04:43:33.863315 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 04:43:33.865212 7975 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:33.864486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48202 (local address 127.1.226.193:34231)
0928 04:43:33.864545 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:33.864547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:33.864590 (+ 43us) server_negotiation.cc:407] Connection header received
0928 04:43:33.864749 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:33.864750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:33.864761 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:33.864832 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:33.865096 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48202: BlockingRecv error: recv got EOF from 127.0.0.1:48202 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 04:43:33.865289 1931 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570585693787-1931-0/krb5kdc/krb5cc' of type FILE
Sep 28 04:43:33 dist-test-slave-j5c3 krb5kdc[7464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:33.866266 1931 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 04:43:33 dist-test-slave-j5c3 krb5kdc[7464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:33.867071 1931 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 04:43:33 dist-test-slave-j5c3 krb5kdc[7464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.193@KRBTEST.COM
I20260928 04:43:33.873919 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:33.867526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48218 (local address 127.1.226.193:34231)
0928 04:43:33.867561 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:33.867562 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:33.867600 (+ 38us) server_negotiation.cc:407] Connection header received
0928 04:43:33.867648 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:33.867649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:33.867656 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:33.867704 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:33.868044 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:33.868579 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:33.869315 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:33.869533 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:33.869590 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:33.871579 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:33.871580 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:33.871581 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:33.871586 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:33.872837 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:33.873141 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:33.873141 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:33.873142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:33.873190 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:33.873410 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:33.873411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:33.873411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:33.873498 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:33.873571 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:33.873799 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 04:43:33.873857 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 04:43:33.874609 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 7571
I20260928 04:43:33.879860 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 7705
I20260928 04:43:33.885120 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 7839
I20260928 04:43:33.889398 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 7477
2026-09-28T04:43:33Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3858 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:43:33 dist-test-slave-j5c3 krb5kdc[7988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:43:33 dist-test-slave-j5c3 krb5kdc[7988](info): set up 2 sockets
Sep 28 04:43:33 dist-test-slave-j5c3 krb5kdc[7988](info): commencing operation
krb5kdc: starting...
W20260928 04:43:34.912504 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:43:34 dist-test-slave-j5c3 krb5kdc[7988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:43:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:43:35Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.254.keytab.
Entry for principal oryx/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.254.keytab.
I20260928 04:43:35.035154 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:42709
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.254.keytab
--principal=oryx/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:42709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:35.115024 8001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:35.115233 8001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:35.115255 8001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:35.116427 8001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:43:35.116472 8001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:35.116487 8001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:35.116498 8001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:43:35.116510 8001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:43:35.118008 8001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:42709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.254.keytab
--principal=oryx/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:42709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:35.118273 8001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:35.118520 8001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:35.123399 8006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:35.123414 8007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:35.123428 8009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:35.123574 8001 server_base.cc:1034] running on GCE node
I20260928 04:43:35.123943 8001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:35.124266 8001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:35.125414 8001 hybrid_clock.cc:648] HybridClock initialized: now 1790570615125395 us; error 41 us; skew 500 ppm
Sep 28 04:43:35 dist-test-slave-j5c3 krb5kdc[7988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570615, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:35.128108 8001 init.cc:377] Logged in from keytab as oryx/127.1.226.254@KRBTEST.COM (short username oryx)
I20260928 04:43:35.128625 8001 webserver.cc:460] Webserver started at http://127.1.226.254:46497/ using document root <none> and password file <none>
I20260928 04:43:35.128909 8001 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:35.128979 8001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:35.129146 8001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:35.130239 8001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "ab16fb1ab1aa488299d5049c1f16fa81"
format_stamp: "Formatted at 2026-09-28 04:43:35 on dist-test-slave-j5c3"
server_key: "35438496615c8a8608b193df151918f8"
server_key_iv: "4f9d4b10c0ae077eb78a3eb6fcd57b17"
server_key_version: "encryptionkey@0"
I20260928 04:43:35.130623 8001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "ab16fb1ab1aa488299d5049c1f16fa81"
format_stamp: "Formatted at 2026-09-28 04:43:35 on dist-test-slave-j5c3"
server_key: "35438496615c8a8608b193df151918f8"
server_key_iv: "4f9d4b10c0ae077eb78a3eb6fcd57b17"
server_key_version: "encryptionkey@0"
I20260928 04:43:35.132439 8001 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20260928 04:43:35.133245 8016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:35.133414 8001 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:43:35.133522 8001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "ab16fb1ab1aa488299d5049c1f16fa81"
format_stamp: "Formatted at 2026-09-28 04:43:35 on dist-test-slave-j5c3"
server_key: "35438496615c8a8608b193df151918f8"
server_key_iv: "4f9d4b10c0ae077eb78a3eb6fcd57b17"
server_key_version: "encryptionkey@0"
I20260928 04:43:35.133610 8001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:35.146204 8001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:35.148559 8001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:35.148736 8001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:35.152733 8001 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:42709
I20260928 04:43:35.152863 8068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:42709 every 8 connection(s)
I20260928 04:43:35.153187 8001 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:43:35.154163 8069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:35.156744 8069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81: Bootstrap starting.
I20260928 04:43:35.157327 8069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:35.157620 8069 log.cc:824] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:35.158347 8069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81: No bootstrap required, opened a new log
I20260928 04:43:35.160023 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 8001
I20260928 04:43:35.160120 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:43:35.160341 1931 external_mini_cluster.cc:1428] Setting key 1f69aebc4b76a0ac229bb9f53f3332d2
I20260928 04:43:35.160678 8069 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } }
I20260928 04:43:35.160889 8069 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:35.160971 8069 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab16fb1ab1aa488299d5049c1f16fa81, State: Initialized, Role: FOLLOWER
I20260928 04:43:35.161152 8069 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } }
I20260928 04:43:35.161230 8069 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:43:35.161252 8069 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:43:35.161329 8069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:35.162168 8069 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } }
I20260928 04:43:35.162298 8069 leader_election.cc:302] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab16fb1ab1aa488299d5049c1f16fa81; no voters:
I20260928 04:43:35.162513 8069 leader_election.cc:288] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:43:35.162539 8074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:35.162647 8074 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [term 1 LEADER]: Becoming Leader. State: Replica: ab16fb1ab1aa488299d5049c1f16fa81, State: Running, Role: LEADER
I20260928 04:43:35.162787 8074 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } }
I20260928 04:43:35.162792 8069 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:43:35.163304 8076 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab16fb1ab1aa488299d5049c1f16fa81. Latest consensus state: current_term: 1 leader_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } }
I20260928 04:43:35.163415 8076 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:35.163300 8075 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } }
I20260928 04:43:35.163587 8075 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:35.163641 8078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:43:35.164276 8078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:43:35.166448 8078 catalog_manager.cc:1347] Generated new cluster ID: 2e45c915021c4b2b9faa0da3abf6c355
I20260928 04:43:35.167269 8078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:43:35 dist-test-slave-j5c3 krb5kdc[7988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM
I20260928 04:43:35.172010 8073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:35.160944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48508 (local address 127.1.226.254:42709)
0928 04:43:35.161136 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:35.161152 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:35.161179 (+ 27us) server_negotiation.cc:407] Connection header received
0928 04:43:35.161338 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:35.161346 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:35.161403 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:35.161534 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:35.162410 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.163641 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.164682 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.165227 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.165354 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:35.168422 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:35.168426 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:35.168427 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:35.168438 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:35.170378 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:35.170776 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:35.170779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:35.170779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:35.170890 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:35.171103 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:35.171104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:35.171104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:35.171384 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:35.171533 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:35.171696 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 04:43:35.171823 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":77,"threads_started":1}
I20260928 04:43:35.198627 8078 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:43:35.199254 8078 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:43:35.202797 8078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81: Generated new TSK 0
I20260928 04:43:35.203009 8078 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.226.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.226.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.226.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.193.keytab.
Entry for principal oryx/127.1.226.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.193.keytab.
I20260928 04:43:35.240130 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:0
--local_ip_for_outbound_sockets=127.1.226.193
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:42709
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.193.keytab
--principal=oryx/127.1.226.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:35.344946 8095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:35.345180 8095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:35.345219 8095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:35.346371 8095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:35.346443 8095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:35.346487 8095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:43:35.347949 8095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.193.keytab
--principal=oryx/127.1.226.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:42709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:35.348243 8095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:35.348476 8095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:35.354120 8100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:35.354116 8101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:35.354112 8103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:35.354327 8095 server_base.cc:1034] running on GCE node
I20260928 04:43:35.354530 8095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:35.354815 8095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:35.355969 8095 hybrid_clock.cc:648] HybridClock initialized: now 1790570615355956 us; error 27 us; skew 500 ppm
Sep 28 04:43:35 dist-test-slave-j5c3 krb5kdc[7988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570615, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:35.358539 8095 init.cc:377] Logged in from keytab as oryx/127.1.226.193@KRBTEST.COM (short username oryx)
I20260928 04:43:35.359009 8095 webserver.cc:460] Webserver started at http://127.1.226.193:40891/ using document root <none> and password file <none>
I20260928 04:43:35.359206 8095 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:35.359251 8095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:35.359336 8095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:35.360378 8095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/data/instance:
uuid: "554061b51b954cdf83cd7da7df877b51"
format_stamp: "Formatted at 2026-09-28 04:43:35 on dist-test-slave-j5c3"
server_key: "d94b070a58d81228fbc17d250ab38b62"
server_key_iv: "472cce503a4c9f63a5906d5bb969cc3d"
server_key_version: "encryptionkey@0"
I20260928 04:43:35.360679 8095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance:
uuid: "554061b51b954cdf83cd7da7df877b51"
format_stamp: "Formatted at 2026-09-28 04:43:35 on dist-test-slave-j5c3"
server_key: "d94b070a58d81228fbc17d250ab38b62"
server_key_iv: "472cce503a4c9f63a5906d5bb969cc3d"
server_key_version: "encryptionkey@0"
I20260928 04:43:35.361938 8095 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:35.362807 8110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:35.362995 8095 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:35.363076 8095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "554061b51b954cdf83cd7da7df877b51"
format_stamp: "Formatted at 2026-09-28 04:43:35 on dist-test-slave-j5c3"
server_key: "d94b070a58d81228fbc17d250ab38b62"
server_key_iv: "472cce503a4c9f63a5906d5bb969cc3d"
server_key_version: "encryptionkey@0"
I20260928 04:43:35.363130 8095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:35.374621 8095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:35.376937 8095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:35.377050 8095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:35.377228 8095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:35.377543 8095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:35.377580 8095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:35.377601 8095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:35.377614 8095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:35.383409 8095 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:43207
I20260928 04:43:35.383514 8223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:43207 every 8 connection(s)
I20260928 04:43:35.383829 8095 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
I20260928 04:43:35.385973 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 8095
I20260928 04:43:35.386097 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance
I20260928 04:43:35.386304 1931 external_mini_cluster.cc:1428] Setting key f3612d2072f23802d1eb570f2099a148
Sep 28 04:43:35 dist-test-slave-j5c3 krb5kdc[7988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570615, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM
I20260928 04:43:35.394187 8073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:35.384616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:39091 (local address 127.1.226.254:42709)
0928 04:43:35.384700 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:35.384702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:35.384885 (+ 183us) server_negotiation.cc:407] Connection header received
0928 04:43:35.385363 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:35.385365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:35.385380 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:35.385436 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:35.386533 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.387150 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.388188 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.388408 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.388495 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:35.391008 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:35.391010 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:35.391010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:35.391021 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:35.392722 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:35.393151 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:35.393153 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:35.393153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:35.393222 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:35.393476 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:35.393478 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:35.393478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:35.393623 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:35.393701 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:35.393976 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 04:43:35.394066 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 04:43:35.394217 8226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:35.384557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.1.226.193:39091)
0928 04:43:35.384757 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:35.384796 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:35.385267 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:35.385509 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:35.385513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:35.386135 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:35.386412 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:35.386419 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.387246 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.387247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:35.388067 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:35.388069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.388507 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.388508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:35.388654 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:35.389281 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:35.389290 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:35.390901 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:35.392834 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:35.392840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:35.392842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:35.393072 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:35.393297 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:35.393298 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:35.393298 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:35.393404 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:35.393696 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:35.393701 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:35.393835 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 04:43:35.393921 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 04:43:35.394047 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for oryx/127.1.226.194@KRBTEST.COM; defaulting to no policy
I20260928 04:43:35.394799 8224 heartbeater.cc:347] Connected to a master server at 127.1.226.254:42709
I20260928 04:43:35.394917 8224 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:35.395116 8224 heartbeater.cc:511] Master 127.1.226.254:42709 requested a full tablet report, sending...
I20260928 04:43:35.395617 8033 ts_manager.cc:194] Registered new tserver with Master: 554061b51b954cdf83cd7da7df877b51 (127.1.226.193:43207)
I20260928 04:43:35.396318 8033 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.226.193@KRBTEST.COM'} at 127.1.226.193:39091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.226.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.226.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.194.keytab.
Entry for principal oryx/127.1.226.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.194.keytab.
I20260928 04:43:35.411235 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:0
--local_ip_for_outbound_sockets=127.1.226.194
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:42709
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.194.keytab
--principal=oryx/127.1.226.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:35.492950 8229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:35.493153 8229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:35.493176 8229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:35.494256 8229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:35.494289 8229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:35.494314 8229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:43:35.495734 8229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.194.keytab
--principal=oryx/127.1.226.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:42709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:35.495926 8229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:35.496116 8229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:35.501394 8237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:35.501426 8234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:35.501459 8229 server_base.cc:1034] running on GCE node
W20260928 04:43:35.501430 8235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:35.501842 8229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:35.502134 8229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:35.503348 8229 hybrid_clock.cc:648] HybridClock initialized: now 1790570615503332 us; error 45 us; skew 500 ppm
Sep 28 04:43:35 dist-test-slave-j5c3 krb5kdc[7988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570615, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:35.506255 8229 init.cc:377] Logged in from keytab as oryx/127.1.226.194@KRBTEST.COM (short username oryx)
I20260928 04:43:35.506752 8229 webserver.cc:460] Webserver started at http://127.1.226.194:35161/ using document root <none> and password file <none>
I20260928 04:43:35.507021 8229 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:35.507107 8229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:35.507292 8229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:35.508594 8229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/data/instance:
uuid: "fc4ea0c4703a4ab998401c9357d616da"
format_stamp: "Formatted at 2026-09-28 04:43:35 on dist-test-slave-j5c3"
server_key: "6a8bf799b5f3011e8694f5a2c29e6717"
server_key_iv: "3293070157c141390f6e6fc303940861"
server_key_version: "encryptionkey@0"
I20260928 04:43:35.508996 8229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance:
uuid: "fc4ea0c4703a4ab998401c9357d616da"
format_stamp: "Formatted at 2026-09-28 04:43:35 on dist-test-slave-j5c3"
server_key: "6a8bf799b5f3011e8694f5a2c29e6717"
server_key_iv: "3293070157c141390f6e6fc303940861"
server_key_version: "encryptionkey@0"
I20260928 04:43:35.510612 8229 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:43:35.511498 8244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:35.511672 8229 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:43:35.511796 8229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "fc4ea0c4703a4ab998401c9357d616da"
format_stamp: "Formatted at 2026-09-28 04:43:35 on dist-test-slave-j5c3"
server_key: "6a8bf799b5f3011e8694f5a2c29e6717"
server_key_iv: "3293070157c141390f6e6fc303940861"
server_key_version: "encryptionkey@0"
I20260928 04:43:35.511881 8229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:35.533818 8229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:35.535852 8229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:35.536057 8229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:35.536439 8229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:35.536830 8229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:35.536896 8229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:35.536944 8229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:35.536976 8229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:35.543179 8229 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:33589
I20260928 04:43:35.543246 8357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:33589 every 8 connection(s)
I20260928 04:43:35.543617 8229 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
I20260928 04:43:35.546569 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 8229
I20260928 04:43:35.546725 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance
I20260928 04:43:35.546919 1931 external_mini_cluster.cc:1428] Setting key 40a1ddb39fd92b34acbedf88e8b44d3d
Sep 28 04:43:35 dist-test-slave-j5c3 krb5kdc[7988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570615, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM
I20260928 04:43:35.554997 8073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:35.544412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:36057 (local address 127.1.226.254:42709)
0928 04:43:35.544467 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:35.544470 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:35.544719 (+ 249us) server_negotiation.cc:407] Connection header received
0928 04:43:35.545238 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:35.545239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:35.545249 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:35.545331 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:35.546369 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.546943 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.548289 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.548675 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.548760 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:35.551622 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:35.551624 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:35.551624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:35.551632 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:35.553257 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:35.553824 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:35.553826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:35.553826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:35.553888 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:35.554234 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:35.554235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:35.554235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:35.554363 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:35.554431 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:35.554842 (+ 411us) server_negotiation.cc:299] Negotiation successful
0928 04:43:35.554926 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:43:35.555259 8360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:35.544413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.1.226.194:36057)
0928 04:43:35.544579 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:35.544620 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:35.545142 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:35.545400 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:35.545404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:35.546030 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:35.546272 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:35.546278 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.547046 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.547047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:35.548207 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:35.548210 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.548769 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.548770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:35.548888 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:35.549541 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:35.549553 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:35.551486 (+ 1933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:35.553358 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:35.553369 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:35.553372 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:35.553721 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:35.553964 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:35.553965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:35.553965 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:35.554123 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:35.554448 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:35.554456 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:35.554657 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 04:43:35.554799 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 04:43:35.554994 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
I20260928 04:43:35.555950 8358 heartbeater.cc:347] Connected to a master server at 127.1.226.254:42709
I20260928 04:43:35.556079 8358 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.1.226.195@KRBTEST.COM; defaulting to no policy
I20260928 04:43:35.556329 8358 heartbeater.cc:511] Master 127.1.226.254:42709 requested a full tablet report, sending...
I20260928 04:43:35.556875 8033 ts_manager.cc:194] Registered new tserver with Master: fc4ea0c4703a4ab998401c9357d616da (127.1.226.194:33589)
I20260928 04:43:35.557322 8033 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.226.194@KRBTEST.COM'} at 127.1.226.194:36057
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.226.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.226.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.195.keytab.
Entry for principal oryx/127.1.226.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.195.keytab.
I20260928 04:43:35.575135 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:0
--local_ip_for_outbound_sockets=127.1.226.195
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:42709
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.195.keytab
--principal=oryx/127.1.226.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:35.656261 8363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:35.656515 8363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:35.656554 8363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:35.657727 8363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:35.657791 8363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:35.657835 8363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:43:35.659348 8363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.195.keytab
--principal=oryx/127.1.226.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:42709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:35.659611 8363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:35.659840 8363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:35.664976 8368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:35.664964 8369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:35.665239 8363 server_base.cc:1034] running on GCE node
W20260928 04:43:35.665356 8371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:35.665546 8363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:35.665750 8363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:35.666888 8363 hybrid_clock.cc:648] HybridClock initialized: now 1790570615666877 us; error 28 us; skew 500 ppm
Sep 28 04:43:35 dist-test-slave-j5c3 krb5kdc[7988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570615, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:35.669395 8363 init.cc:377] Logged in from keytab as oryx/127.1.226.195@KRBTEST.COM (short username oryx)
I20260928 04:43:35.669775 8363 webserver.cc:460] Webserver started at http://127.1.226.195:44397/ using document root <none> and password file <none>
I20260928 04:43:35.670030 8363 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:35.670101 8363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:35.670257 8363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:35.671320 8363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/data/instance:
uuid: "033aee8210cc40bf922b90e5199137c4"
format_stamp: "Formatted at 2026-09-28 04:43:35 on dist-test-slave-j5c3"
server_key: "129ff101f8d91300ff0cce2a311e3fdc"
server_key_iv: "e99204d91a432189a2f5187c4e75af47"
server_key_version: "encryptionkey@0"
I20260928 04:43:35.671685 8363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance:
uuid: "033aee8210cc40bf922b90e5199137c4"
format_stamp: "Formatted at 2026-09-28 04:43:35 on dist-test-slave-j5c3"
server_key: "129ff101f8d91300ff0cce2a311e3fdc"
server_key_iv: "e99204d91a432189a2f5187c4e75af47"
server_key_version: "encryptionkey@0"
I20260928 04:43:35.673098 8363 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:43:35.673835 8378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:35.674006 8363 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:43:35.674111 8363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "033aee8210cc40bf922b90e5199137c4"
format_stamp: "Formatted at 2026-09-28 04:43:35 on dist-test-slave-j5c3"
server_key: "129ff101f8d91300ff0cce2a311e3fdc"
server_key_iv: "e99204d91a432189a2f5187c4e75af47"
server_key_version: "encryptionkey@0"
I20260928 04:43:35.674192 8363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:35.681116 8363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:35.683099 8363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:35.683266 8363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:35.683491 8363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:35.683799 8363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:35.683853 8363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:35.683895 8363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:35.683923 8363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:35.689778 8363 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:45509
I20260928 04:43:35.689859 8491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:45509 every 8 connection(s)
I20260928 04:43:35.690191 8363 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
Sep 28 04:43:35 dist-test-slave-j5c3 krb5kdc[7988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570615, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM
I20260928 04:43:35.699590 8073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:35.690935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:41977 (local address 127.1.226.254:42709)
0928 04:43:35.691019 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:35.691022 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:35.691182 (+ 160us) server_negotiation.cc:407] Connection header received
0928 04:43:35.691638 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:35.691639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:35.691649 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:35.691703 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:35.692680 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.693197 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.694159 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.694342 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.694398 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:35.696572 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:35.696573 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:35.696573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:35.696580 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:35.698144 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:35.698562 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:35.698564 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:35.698564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:35.698628 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:35.698973 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:35.698974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:35.698974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:35.699102 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:35.699170 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:35.699448 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 04:43:35.699523 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 04:43:35.699765 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:35.690898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.1.226.195:41977)
0928 04:43:35.691059 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:35.691095 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:35.691557 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:35.691756 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:35.691760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:35.692394 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:35.692584 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:35.692589 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.693283 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.693284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:35.694066 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:35.694068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.694416 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.694417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:35.694512 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:35.694960 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:35.694968 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:35.696465 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:35.698229 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:35.698235 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:35.698237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:35.698466 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:35.698766 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:35.698766 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:35.698767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:35.698881 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:35.699170 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:35.699174 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:35.699301 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 04:43:35.699386 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 04:43:35.699510 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":97,"threads_started":1}
I20260928 04:43:35.700364 8492 heartbeater.cc:347] Connected to a master server at 127.1.226.254:42709
I20260928 04:43:35.700454 8492 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:35.700456 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 8363
I20260928 04:43:35.700546 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance
I20260928 04:43:35.700680 8492 heartbeater.cc:511] Master 127.1.226.254:42709 requested a full tablet report, sending...
I20260928 04:43:35.700724 1931 external_mini_cluster.cc:1428] Setting key 38b5db2bd2f3392ad526e4001b3415f6
I20260928 04:43:35.701107 8032 ts_manager.cc:194] Registered new tserver with Master: 033aee8210cc40bf922b90e5199137c4 (127.1.226.195:45509)
I20260928 04:43:35.701558 8032 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.226.195@KRBTEST.COM'} at 127.1.226.195:41977
I20260928 04:43:35.701642 1931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.226.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.226.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.226.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.253.keytab.
Entry for principal oryx/127.1.226.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.253.keytab.
I20260928 04:43:35.728932 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.253:40789
--webserver_interface=127.1.226.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.253.keytab
--principal=oryx/127.1.226.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:42709,127.1.226.253:40789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:35.816197 8497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:35.816426 8497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:35.816450 8497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:35.817688 8497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:43:35.817739 8497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:35.817756 8497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:35.817817 8497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:43:35.817849 8497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:43:35.819420 8497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:42709,127.1.226.253:40789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.253.keytab
--principal=oryx/127.1.226.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.253:40789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.226.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
W20260928 04:43:35.819569 8497 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.226.254:42709,127.1.226.253:40789'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 04:43:35.819739 8497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:35.819988 8497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:35.824922 8502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:35.824990 8505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:35.825085 8497 server_base.cc:1034] running on GCE node
W20260928 04:43:35.824986 8503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:35.825445 8497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:35.825744 8497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:35.826910 8497 hybrid_clock.cc:648] HybridClock initialized: now 1790570615826889 us; error 38 us; skew 500 ppm
Sep 28 04:43:35 dist-test-slave-j5c3 krb5kdc[7988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570615, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:35.829690 8497 init.cc:377] Logged in from keytab as oryx/127.1.226.253@KRBTEST.COM (short username oryx)
I20260928 04:43:35.830155 8497 webserver.cc:460] Webserver started at http://127.1.226.253:44219/ using document root <none> and password file <none>
I20260928 04:43:35.830349 8497 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:35.830389 8497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:35.830482 8497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:35.831470 8497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-1/data/instance:
uuid: "8030293b08fe463b9af2092f0c6a7233"
format_stamp: "Formatted at 2026-09-28 04:43:35 on dist-test-slave-j5c3"
server_key: "3d2da762c61202f5597263068ba1ce2b"
server_key_iv: "21600ba445cf39c7a8072880e67dd0f7"
server_key_version: "encryptionkey@0"
I20260928 04:43:35.831776 8497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-1/wal/instance:
uuid: "8030293b08fe463b9af2092f0c6a7233"
format_stamp: "Formatted at 2026-09-28 04:43:35 on dist-test-slave-j5c3"
server_key: "3d2da762c61202f5597263068ba1ce2b"
server_key_iv: "21600ba445cf39c7a8072880e67dd0f7"
server_key_version: "encryptionkey@0"
I20260928 04:43:35.833129 8497 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:43:35.833875 8512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:35.834048 8497 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:43:35.834151 8497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-1/wal
uuid: "8030293b08fe463b9af2092f0c6a7233"
format_stamp: "Formatted at 2026-09-28 04:43:35 on dist-test-slave-j5c3"
server_key: "3d2da762c61202f5597263068ba1ce2b"
server_key_iv: "21600ba445cf39c7a8072880e67dd0f7"
server_key_version: "encryptionkey@0"
I20260928 04:43:35.834227 8497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:35.857906 8497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:35.860102 8497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:35.860316 8497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:35.864789 8497 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.253:40789
I20260928 04:43:35.864845 8564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.253:40789 every 8 connection(s)
I20260928 04:43:35.865211 8497 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-1/data/info.pb
I20260928 04:43:35.866267 8565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:35.867862 8565 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 04:43:35 dist-test-slave-j5c3 krb5kdc[7988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570615, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.253@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM
I20260928 04:43:35.874147 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 8497
I20260928 04:43:35.878134 8073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:35.868616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40436 (local address 127.1.226.254:42709)
0928 04:43:35.868694 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:35.868696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:35.868932 (+ 236us) server_negotiation.cc:407] Connection header received
0928 04:43:35.869379 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:35.869380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:35.869391 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:35.869452 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:35.870110 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.870626 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.871713 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.871916 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.871988 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:35.874555 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:35.874557 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:35.874558 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:35.874567 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:35.876382 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:35.876867 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:35.876868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:35.876868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:35.876970 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:35.877252 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:35.877253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:35.877253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:35.877470 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:35.877545 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:35.877934 (+ 389us) server_negotiation.cc:299] Negotiation successful
0928 04:43:35.878002 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 04:43:35.878340 8567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:35.868599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.0.0.1:40436)
0928 04:43:35.868808 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:35.868844 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:35.869267 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:35.869507 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:35.869510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:35.869752 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:35.869945 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:35.869949 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.870721 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.870722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:35.871551 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:35.871553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.872000 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.872001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:35.872103 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:35.872760 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:35.872769 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:35.874386 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:35.876463 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:35.876468 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:35.876470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:35.876785 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:35.877048 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:35.877049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:35.877049 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:35.877171 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:35.877567 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:35.877572 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:35.877758 (+ 186us) client_negotiation.cc:769] Sending connection context
0928 04:43:35.877906 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 04:43:35.878106 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":136,"threads_started":1}
I20260928 04:43:35.879001 8565 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 04:43:35 dist-test-slave-j5c3 krb5kdc[7988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.226.253@KRBTEST.COM
Sep 28 04:43:35 dist-test-slave-j5c3 krb5kdc[7988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570615, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.253@KRBTEST.COM for oryx/127.1.226.253@KRBTEST.COM
I20260928 04:43:35.884539 8569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:35.874711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33938 (local address 127.1.226.253:40789)
0928 04:43:35.874934 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:35.874939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:35.874967 (+ 28us) server_negotiation.cc:407] Connection header received
0928 04:43:35.874992 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:35.874995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:35.875012 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:35.875150 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:35.875893 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.876969 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.877886 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.878346 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.878423 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:35.881120 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:35.881130 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:35.881130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:35.881138 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:35.883190 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:35.883548 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:35.883552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:35.883552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:35.883613 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:35.883872 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:35.883873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:35.883873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:35.884061 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:35.884254 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:35.884347 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 04:43:35.884407 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":113,"threads_started":1}
I20260928 04:43:35.886633 8567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:35.879275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:40789 (local address 127.0.0.1:33952)
0928 04:43:35.879325 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:35.879335 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:35.879370 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:35.879590 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:35.879591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:35.879741 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:35.879778 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:35.879779 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.880747 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.880748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:35.881258 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:35.881260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.881727 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.881728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:35.881828 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:35.882196 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:35.882199 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:35.883477 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:35.885859 (+ 2382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:35.885860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:35.885860 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:35.886012 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:35.886203 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:35.886203 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:35.886204 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:35.886250 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:35.886421 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:35.886421 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:35.886453 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 04:43:35.886502 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 04:43:35.886574 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 04:43:35.886783 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:35.879331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.1.226.253:40789)
0928 04:43:35.879490 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:35.879492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:35.879497 (+ 5us) server_negotiation.cc:407] Connection header received
0928 04:43:35.879511 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:35.879511 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:35.879518 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:35.879589 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:35.879852 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.880648 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.881369 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.881648 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.881727 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:35.883591 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:35.883597 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:35.883598 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:35.883603 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:35.885790 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:35.886075 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:35.886076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:35.886076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:35.886123 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:35.886292 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:35.886292 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:35.886293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:35.886376 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:35.886420 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:35.886516 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 04:43:35.886616 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
I20260928 04:43:35.888033 8565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233: Bootstrap starting.
I20260928 04:43:35.888695 8565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:35.888974 8565 log.cc:824] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:35.889663 8565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233: No bootstrap required, opened a new log
I20260928 04:43:35.891011 8565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } }
I20260928 04:43:35.891139 8565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:35.891201 8565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8030293b08fe463b9af2092f0c6a7233, State: Initialized, Role: FOLLOWER
I20260928 04:43:35.891326 8565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } }
I20260928 04:43:35.891614 8565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:43:35.891580 8572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } } }
I20260928 04:43:35.891724 8572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 04:43:35.893345 8583 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:43:35.893422 8583 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:43:35.895013 8497 master_runner.cc:428] Detected that this master 8030293b08fe463b9af2092f0c6a7233 is joining an existing cluster
I20260928 04:43:35.895064 8497 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.226.253:40789
I20260928 04:43:35.902897 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:35.895920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33960 (local address 127.1.226.253:40789)
0928 04:43:35.895965 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:35.895968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:35.896220 (+ 252us) server_negotiation.cc:407] Connection header received
0928 04:43:35.896271 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:35.896272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:35.896282 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:35.896341 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:35.896697 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.897442 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.898562 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.898841 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.898913 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:35.900116 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:35.900117 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:35.900117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:35.900124 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:35.901806 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:35.902133 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:35.902134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:35.902134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:35.902182 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:35.902388 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:35.902389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:35.902389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:35.902501 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:35.902574 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:35.902704 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 04:43:35.902788 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:43:35.902906 8073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:35.895980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40438 (local address 127.1.226.254:42709)
0928 04:43:35.896021 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:35.896023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:35.896143 (+ 120us) server_negotiation.cc:407] Connection header received
0928 04:43:35.896272 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:35.896273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:35.896284 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:35.896315 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:35.896765 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.897483 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.898368 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.898643 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.898706 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:35.899778 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:35.899780 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:35.899781 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:35.899797 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:35.901536 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:35.902040 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:35.902041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:35.902041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:35.902118 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:35.902328 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:35.902329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:35.902329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:35.902466 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:35.902510 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:35.902725 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 04:43:35.902809 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:43:35.902982 8590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:35.895950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:40789 (local address 127.0.0.1:33960)
0928 04:43:35.896131 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:35.896141 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:35.896215 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:35.896343 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:35.896343 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:35.896572 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:35.896627 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:35.896628 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.897530 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.897531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:35.898470 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:35.898472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.898912 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.898913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:35.899221 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:35.899625 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:35.899629 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:35.900033 (+ 404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:35.901892 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:35.901892 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:35.901893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:35.902054 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:35.902253 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:35.902254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:35.902255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:35.902327 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:35.902566 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:35.902567 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:35.902606 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 04:43:35.902685 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 04:43:35.902846 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":105,"thread_start_us":143,"threads_started":1}
I20260928 04:43:35.903885 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:35.895913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.0.0.1:40438)
0928 04:43:35.896078 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:35.896089 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:35.896195 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:35.896378 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:35.896379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:35.896632 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:35.896696 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:35.896698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.897563 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.897564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:35.898285 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:35.898287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.898718 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.898719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:35.898801 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:35.899221 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:35.899226 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:35.899686 (+ 460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:35.901634 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:35.901635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:35.901635 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:35.901942 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:35.902199 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:35.902200 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:35.902200 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:35.902260 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:35.902555 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:35.902556 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:35.902595 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 04:43:35.903665 (+ 1070us) client_negotiation.cc:240] Negotiation successful
0928 04:43:35.903778 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":47,"thread_start_us":83,"threads_started":1}
Sep 28 04:43:35 dist-test-slave-j5c3 krb5kdc[7988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570615, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.254@KRBTEST.COM for oryx/127.1.226.253@KRBTEST.COM
I20260928 04:43:35.914265 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:35.906156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33974 (local address 127.1.226.253:40789)
0928 04:43:35.906195 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:35.906197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:35.906407 (+ 210us) server_negotiation.cc:407] Connection header received
0928 04:43:35.906490 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:35.906491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:35.906499 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:35.906545 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:35.907182 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.907684 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.908855 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.909023 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.909076 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:35.911453 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:35.911455 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:35.911455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:35.911464 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:35.912942 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:35.913381 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:35.913382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:35.913383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:35.913448 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:35.913666 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:35.913668 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:35.913668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:35.913791 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:35.913875 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:35.914071 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 04:43:35.914160 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:43:35.914309 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:35.906129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:40789 (local address 127.0.0.1:33974)
0928 04:43:35.906287 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:35.906338 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:35.906416 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:35.906550 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:35.906553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:35.907040 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:35.907104 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:35.907107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.907762 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.907763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:35.908778 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:35.908780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.909095 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.909096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:35.909188 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:35.909666 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:35.909672 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:35.911336 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:35.913051 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:35.913059 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:35.913062 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:35.913286 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:35.913520 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:35.913522 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:35.913522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:35.913599 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:35.913874 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:35.913883 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:35.913938 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 04:43:35.914044 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 04:43:35.914174 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260928 04:43:35.915057 8032 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.226.253:40789: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8030293b08fe463b9af2092f0c6a7233"
member_type: NON_VOTER
last_known_addr {
host: "127.1.226.253"
port: 40789
}
attrs {
promote: true
}
}
dest_uuid: "ab16fb1ab1aa488299d5049c1f16fa81"
cas_config_opid_index: -1
I20260928 04:43:35.915411 8032 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: NON_VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: true } }
I20260928 04:43:35.916077 8592 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8030293b08fe463b9af2092f0c6a7233 (127.1.226.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: NON_VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: true } } }
I20260928 04:43:35.916448 8539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:35.917272 8592 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.226.253:40789
I20260928 04:43:35.917368 8539 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [term 1 FOLLOWER]: Refusing update from remote peer ab16fb1ab1aa488299d5049c1f16fa81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 04:43:35.917443 8593 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: NON_VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: true } } }
I20260928 04:43:35.917586 8497 master.cc:537] Master@127.1.226.253:40789 shutting down...
I20260928 04:43:35.917606 8593 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:35.917737 8593 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: NON_VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 04:43:35.918499 8020 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 04:43:35.918946 8020 consensus_peers.cc:487] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 -> Peer 8030293b08fe463b9af2092f0c6a7233 (127.1.226.253:40789): Couldn't send request to peer 8030293b08fe463b9af2092f0c6a7233. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 04:43:35.920073 8497 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:43:35.920199 8497 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:35.920252 8497 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233: stopping tablet replica
I20260928 04:43:35.933111 8497 master.cc:552] Master@127.1.226.253:40789 shutdown complete.
I20260928 04:43:35.933162 8497 master_runner.cc:305] Clearing existing system tablet
I20260928 04:43:35.933319 8497 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:35.933369 8497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:35.934012 8497 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.002s
I20260928 04:43:35.934657 8597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:35.934770 8497 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:35.934823 8497 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:43:35.934846 8497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-1/wal
uuid: "8030293b08fe463b9af2092f0c6a7233"
format_stamp: "Formatted at 2026-09-28 04:43:35 on dist-test-slave-j5c3"
server_key: "3d2da762c61202f5597263068ba1ce2b"
server_key_iv: "21600ba445cf39c7a8072880e67dd0f7"
server_key_version: "encryptionkey@0"
I20260928 04:43:35.935539 8497 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:35.937105 8497 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 04:43:35.937182 8497 log.cc:1196] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 04:43:35.937381 8497 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233: Deleting consensus metadata
I20260928 04:43:35.937525 8497 master_runner.cc:315] Copying system tablet from 127.1.226.254:42709
I20260928 04:43:35.938145 8497 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233: tablet copy: Beginning tablet copy session from remote peer at address 127.1.226.254:42709
I20260928 04:43:35.944308 8073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:35.938556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40452 (local address 127.1.226.254:42709)
0928 04:43:35.938615 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:35.938617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:35.938703 (+ 86us) server_negotiation.cc:407] Connection header received
0928 04:43:35.938785 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:35.938786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:35.938797 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:35.938846 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:35.939167 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.939688 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.940483 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.940760 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.940840 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:35.941770 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:35.941772 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:35.941772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:35.941779 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:35.943113 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:35.943469 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:35.943471 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:35.943471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:35.943528 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:35.943722 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:35.943723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:35.943724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:35.943842 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:35.943915 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:35.944066 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 04:43:35.944145 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 04:43:35.944319 8603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:35.938494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.0.0.1:40452)
0928 04:43:35.938613 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:35.938626 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:35.938720 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:35.938860 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:35.938861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:35.939031 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:35.939089 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:35.939090 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.939776 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.939778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:35.940384 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:35.940387 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.940839 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.940840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:35.940910 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:35.941389 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:35.941393 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:35.941682 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:35.943206 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:35.943207 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:35.943207 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:35.943384 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:35.943600 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:35.943600 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:35.943601 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:35.943661 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:35.943915 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:35.943916 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:35.943961 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 04:43:35.944042 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 04:43:35.944146 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
I20260928 04:43:35.944764 8053 tablet_copy_service.cc:140] P ab16fb1ab1aa488299d5049c1f16fa81: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8030293b08fe463b9af2092f0c6a7233 ({username='oryx', principal='oryx/127.1.226.253@KRBTEST.COM'} at 127.0.0.1:40452)
I20260928 04:43:35.944877 8053 tablet_copy_service.cc:161] P ab16fb1ab1aa488299d5049c1f16fa81: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8030293b08fe463b9af2092f0c6a7233 at {username='oryx', principal='oryx/127.1.226.253@KRBTEST.COM'} at 127.0.0.1:40452: session id = 8030293b08fe463b9af2092f0c6a7233-00000000000000000000000000000000
I20260928 04:43:35.945696 8053 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 04:43:35.946360 8497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 04:43:35.947458 8497 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233: tablet copy: Starting download of 0 data blocks...
I20260928 04:43:35.947618 8497 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233: tablet copy: Starting download of 1 WAL segments...
I20260928 04:43:35.948518 8497 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 04:43:35.949275 8053 tablet_copy_service.cc:342] P ab16fb1ab1aa488299d5049c1f16fa81: Request end of tablet copy session 8030293b08fe463b9af2092f0c6a7233-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.226.253@KRBTEST.COM'} at 127.0.0.1:40452
I20260928 04:43:35.949348 8053 tablet_copy_service.cc:433] P ab16fb1ab1aa488299d5049c1f16fa81: ending tablet copy session 8030293b08fe463b9af2092f0c6a7233-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8030293b08fe463b9af2092f0c6a7233
I20260928 04:43:35.951658 8497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:35.952940 8604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:35.952951 8607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:35.953184 8605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:35.953382 8497 server_base.cc:1034] running on GCE node
I20260928 04:43:35.953485 8497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:35.953634 8497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:35.954735 8497 hybrid_clock.cc:648] HybridClock initialized: now 1790570615954726 us; error 27 us; skew 500 ppm
Sep 28 04:43:35 dist-test-slave-j5c3 krb5kdc[7988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570615, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:35.956027 8497 init.cc:377] Logged in from keytab as oryx/127.1.226.253@KRBTEST.COM (short username oryx)
I20260928 04:43:35.956369 8497 webserver.cc:460] Webserver started at http://127.1.226.253:46065/ using document root <none> and password file <none>
I20260928 04:43:35.956511 8497 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:35.956552 8497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:35.957155 8497 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:43:35.957732 8614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:35.957854 8497 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:43:35.957899 8497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-1/wal
uuid: "8030293b08fe463b9af2092f0c6a7233"
format_stamp: "Formatted at 2026-09-28 04:43:35 on dist-test-slave-j5c3"
server_key: "3d2da762c61202f5597263068ba1ce2b"
server_key_iv: "21600ba445cf39c7a8072880e67dd0f7"
server_key_version: "encryptionkey@0"
I20260928 04:43:35.957957 8497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:35.965301 8497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:35.967057 8497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:35.970161 8497 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.253:40789
I20260928 04:43:35.970546 8666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.253:40789 every 8 connection(s)
I20260928 04:43:35.970630 8497 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-1/data/info.pb
I20260928 04:43:35.971424 8667 sys_catalog.cc:263] Verifying existing consensus state
I20260928 04:43:35.971839 8667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233: Bootstrap starting.
I20260928 04:43:35.974080 8667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:43:35.974304 8667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233: Bootstrap complete.
I20260928 04:43:35.974509 8667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: NON_VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: true } }
I20260928 04:43:35.974573 8667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8030293b08fe463b9af2092f0c6a7233, State: Initialized, Role: LEARNER
I20260928 04:43:35.974599 8667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: NON_VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: true } }
I20260928 04:43:35.974756 8667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:43:35.974778 8670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: NON_VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: true } } }
I20260928 04:43:35.974886 8670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [sys.catalog]: This master's current role is: LEARNER
I20260928 04:43:35.976130 8681 catalog_manager.cc:1259] Loaded cluster ID: 2e45c915021c4b2b9faa0da3abf6c355
I20260928 04:43:35.976220 8681 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233: loading cluster ID for follower catalog manager: success
I20260928 04:43:35.977595 8681 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233: acquiring CA information for follower catalog manager: success
I20260928 04:43:35.977861 8681 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 04:43:35.992677 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:35.986285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34002 (local address 127.1.226.253:40789)
0928 04:43:35.986409 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:35.986416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:35.986423 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:43:35.986443 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:35.986444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:35.986453 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:35.986523 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:35.987077 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.987718 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.988511 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:35.988717 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:35.988795 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:35.990053 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:35.990054 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:35.990055 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:35.990062 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:35.991465 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:35.991816 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:35.991817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:35.991818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:35.991873 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:35.992071 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:35.992072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:35.992072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:35.992232 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:35.992377 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:35.992473 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 04:43:35.992559 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
WARNING: no policy specified for oryx/127.1.226.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.226.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.226.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.252.keytab.
Entry for principal oryx/127.1.226.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.252.keytab.
I20260928 04:43:36.021709 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.252:45505
--webserver_interface=127.1.226.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.252.keytab
--principal=oryx/127.1.226.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:42709,127.1.226.253:40789,127.1.226.252:45505
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:36.114125 8685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:36.114347 8685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:36.114369 8685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:36.115562 8685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:43:36.115603 8685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:36.115617 8685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:36.115630 8685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:43:36.115643 8685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:43:36.117189 8685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:42709,127.1.226.253:40789,127.1.226.252:45505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.252.keytab
--principal=oryx/127.1.226.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.252:45505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.226.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:36.117393 8685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:36.117568 8685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:36.122238 8693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:36.122414 8685 server_base.cc:1034] running on GCE node
W20260928 04:43:36.122260 8691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:36.122246 8690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:36.122802 8685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:36.123045 8685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:36.124249 8685 hybrid_clock.cc:648] HybridClock initialized: now 1790570616124227 us; error 38 us; skew 500 ppm
Sep 28 04:43:36 dist-test-slave-j5c3 krb5kdc[7988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570616, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:36.126796 8685 init.cc:377] Logged in from keytab as oryx/127.1.226.252@KRBTEST.COM (short username oryx)
I20260928 04:43:36.127242 8685 webserver.cc:460] Webserver started at http://127.1.226.252:45021/ using document root <none> and password file <none>
I20260928 04:43:36.127434 8685 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:36.127475 8685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:36.127562 8685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:36.128717 8685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-2/data/instance:
uuid: "2aec508a2bbf4925857d0d1a1d010956"
format_stamp: "Formatted at 2026-09-28 04:43:36 on dist-test-slave-j5c3"
server_key: "4149979846eb323fd3e322a00445855e"
server_key_iv: "5dbc4fba9eb4abd749d10caf6a020579"
server_key_version: "encryptionkey@0"
I20260928 04:43:36.129040 8685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-2/wal/instance:
uuid: "2aec508a2bbf4925857d0d1a1d010956"
format_stamp: "Formatted at 2026-09-28 04:43:36 on dist-test-slave-j5c3"
server_key: "4149979846eb323fd3e322a00445855e"
server_key_iv: "5dbc4fba9eb4abd749d10caf6a020579"
server_key_version: "encryptionkey@0"
I20260928 04:43:36.130354 8685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:43:36.131085 8700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:36.131351 8685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:36.131425 8685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-2/wal
uuid: "2aec508a2bbf4925857d0d1a1d010956"
format_stamp: "Formatted at 2026-09-28 04:43:36 on dist-test-slave-j5c3"
server_key: "4149979846eb323fd3e322a00445855e"
server_key_iv: "5dbc4fba9eb4abd749d10caf6a020579"
server_key_version: "encryptionkey@0"
I20260928 04:43:36.131510 8685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:36.148388 8685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:36.150753 8685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:36.150914 8685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:36.155169 8685 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.252:45505
I20260928 04:43:36.155234 8752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.252:45505 every 8 connection(s)
I20260928 04:43:36.155579 8685 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-2/data/info.pb
I20260928 04:43:36.157004 8753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:36.157605 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 8685
I20260928 04:43:36.159370 8753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 04:43:36 dist-test-slave-j5c3 krb5kdc[7988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570616, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.252@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM
Sep 28 04:43:36 dist-test-slave-j5c3 krb5kdc[7988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.226.252@KRBTEST.COM
I20260928 04:43:36.168511 8073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.160232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40462 (local address 127.1.226.254:42709)
0928 04:43:36.160300 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.160302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.160318 (+ 16us) server_negotiation.cc:407] Connection header received
0928 04:43:36.160341 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.160342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.160350 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.160399 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:36.161000 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.161776 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.162873 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.163124 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.163202 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.165415 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:36.165417 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:36.165417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:36.165424 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:36.167016 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.167450 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.167452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.167452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.167518 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.167784 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.167785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.167785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.167897 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:36.167947 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.168298 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.168395 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 04:43:36.168654 8756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.160003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.0.0.1:40462)
0928 04:43:36.160190 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:36.160230 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:36.160294 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:36.160442 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:36.160444 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:36.160803 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:36.160910 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.160915 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.161896 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.161897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.162798 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.162799 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.163231 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.163232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.163304 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.163723 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:36.163731 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:36.165337 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:36.167112 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.167118 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.167120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.167373 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.167583 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.167584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.167584 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.167719 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.167958 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:36.167964 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:36.168136 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 04:43:36.168275 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 04:43:36.168420 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":102,"threads_started":1}
I20260928 04:43:36.168854 8755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.158217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37752 (local address 127.1.226.252:45505)
0928 04:43:36.158387 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.158396 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.158421 (+ 25us) server_negotiation.cc:407] Connection header received
0928 04:43:36.158885 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.158892 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.158973 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.159121 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:36.159741 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.161079 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.162061 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.162673 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.162788 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.165899 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:36.165906 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:36.165906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:36.165914 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:36.167650 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.168018 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.168026 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.168026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.168073 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.168287 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.168288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.168288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.168447 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:36.168588 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.168691 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.168762 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20260928 04:43:36.169466 8753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 04:43:36 dist-test-slave-j5c3 krb5kdc[7988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570616, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.252@KRBTEST.COM for oryx/127.1.226.253@KRBTEST.COM
I20260928 04:43:36.176035 8756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.169702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:40789 (local address 127.0.0.1:34010)
0928 04:43:36.169747 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:36.169758 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:36.169796 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:36.169925 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:36.169925 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:36.170031 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:36.170071 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.170072 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.170734 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.170735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.171232 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.171233 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.171633 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.171634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.171710 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.172211 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:36.172216 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:36.173430 (+ 1214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:36.174950 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.174951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.174951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.175200 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.175430 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.175431 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.175432 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.175493 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.175712 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:36.175712 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:36.175749 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 04:43:36.175859 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 04:43:36.175949 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 04:43:36.176041 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.169771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34010 (local address 127.1.226.253:40789)
0928 04:43:36.169828 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.169830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.169838 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:43:36.169861 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.169862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.169874 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.169924 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:36.170188 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.170659 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.171308 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.171510 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.171599 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.173515 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:36.173525 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:36.173525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:36.173532 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:36.174870 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.175270 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.175271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.175271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.175325 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.175557 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.175558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.175558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.175655 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:36.175704 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.175899 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.175969 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 04:43:36.176559 8753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45505 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 04:43:36 dist-test-slave-j5c3 krb5kdc[7988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570616, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.252@KRBTEST.COM for oryx/127.1.226.252@KRBTEST.COM
I20260928 04:43:36.182863 8755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.176914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37768 (local address 127.1.226.252:45505)
0928 04:43:36.176949 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.176951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.176958 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:43:36.177013 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.177014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.177023 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.177074 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:36.177304 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.177847 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.178504 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.178713 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.178803 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.180506 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:36.180507 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:36.180508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:36.180514 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:36.181728 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.182064 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.182065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.182066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.182118 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.182323 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.182324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.182324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.182455 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:36.182519 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.182666 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.182780 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":19}
I20260928 04:43:36.182863 8756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.176875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:45505 (local address 127.0.0.1:37768)
0928 04:43:36.176908 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:36.176918 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:36.176958 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:36.177077 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:36.177077 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:36.177201 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:36.177226 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.177227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.177949 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.177950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.178413 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.178414 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.178795 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.178796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.178840 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.179191 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:36.179194 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:36.180411 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:36.181829 (+ 1418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.181830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.181830 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.181985 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.182188 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.182189 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.182189 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.182240 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.182540 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:36.182540 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:36.182572 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 04:43:36.182632 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 04:43:36.182730 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 04:43:36.184815 8753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956: Bootstrap starting.
I20260928 04:43:36.185500 8753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:36.185796 8753 log.cc:824] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:36.186568 8753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956: No bootstrap required, opened a new log
I20260928 04:43:36.188017 8753 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45505 } }
I20260928 04:43:36.188184 8753 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:36.188263 8753 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aec508a2bbf4925857d0d1a1d010956, State: Initialized, Role: FOLLOWER
I20260928 04:43:36.188400 8753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45505 } }
I20260928 04:43:36.188699 8758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45505 } } }
I20260928 04:43:36.188759 8753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:43:36.188783 8758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 04:43:36.190408 8769 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:43:36.190461 8769 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:43:36.192945 8685 master_runner.cc:428] Detected that this master 2aec508a2bbf4925857d0d1a1d010956 is joining an existing cluster
I20260928 04:43:36.192989 8685 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.226.252:45505
I20260928 04:43:36.201553 8775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.194173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:40789 (local address 127.0.0.1:34020)
0928 04:43:36.194462 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:36.194472 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:36.194561 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:36.194769 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:36.194769 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:36.195062 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:36.195160 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.195161 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.196128 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.196129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.197151 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.197153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.197684 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.197686 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.197771 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.198274 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:36.198279 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:36.198637 (+ 358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:36.200535 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.200536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.200537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.200749 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.200968 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.200969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.200969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.201024 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.201288 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:36.201288 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:36.201326 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 04:43:36.201390 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 04:43:36.201473 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":46,"thread_start_us":55,"threads_started":1}
I20260928 04:43:36.201615 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.194246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34020 (local address 127.1.226.253:40789)
0928 04:43:36.194308 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.194310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.194522 (+ 212us) server_negotiation.cc:407] Connection header received
0928 04:43:36.194643 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.194645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.194657 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.194706 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:36.195255 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.196029 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.197253 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.197548 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.197592 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.198733 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:36.198734 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:36.198735 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:36.198742 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:36.200428 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.200832 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.200833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.200834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.200898 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.201081 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.201082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.201083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.201223 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:36.201294 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.201434 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.201518 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 04:43:36.220444 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 8095
I20260928 04:43:36.225229 8755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.194344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37772 (local address 127.1.226.252:45505)
0928 04:43:36.194377 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.194379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.194713 (+ 334us) server_negotiation.cc:407] Connection header received
0928 04:43:36.194800 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.194800 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.194808 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.194834 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:36.195762 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.196499 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.197470 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.197747 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.197784 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.222267 (+ 24483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:36.222268 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:36.222268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:36.222275 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:36.224092 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.224498 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.224499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.224499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.224544 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.224709 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.224709 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.224710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.224848 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:36.224888 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.225055 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.225130 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 04:43:36.225769 8776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.194345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:45505 (local address 127.0.0.1:37772)
0928 04:43:36.194668 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:36.194676 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:36.194731 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:36.194890 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:36.194891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:36.195170 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:36.195673 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.195675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.196582 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.196583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.197379 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.197381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.197865 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.197866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.197922 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.221254 (+ 23332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:36.221260 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:36.222138 (+ 878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:36.224227 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.224228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.224228 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.224425 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.224605 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.224605 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.224606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.224651 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.224925 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:36.224926 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:36.224960 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 04:43:36.225567 (+ 607us) client_negotiation.cc:240] Negotiation successful
0928 04:43:36.225669 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":64,"threads_started":1}
I20260928 04:43:36.227100 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:43207
--local_ip_for_outbound_sockets=127.1.226.193
--tserver_master_addrs=127.1.226.254:42709
--webserver_port=40891
--webserver_interface=127.1.226.193
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.193.keytab
--principal=oryx/127.1.226.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.226.254:42709,127.1.226.253:40789
--tserver_master_addrs=127.1.226.254:42709,127.1.226.253:40789,127.1.226.252:45505 with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:43:36.227454 8073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.194007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40472 (local address 127.1.226.254:42709)
0928 04:43:36.194063 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.194065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.194536 (+ 471us) server_negotiation.cc:407] Connection header received
0928 04:43:36.194599 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.194601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.194613 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.194664 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:36.195263 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.196025 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.197264 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.197557 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.197592 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.224089 (+ 26497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:36.224090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:36.224091 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:36.224100 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:36.225946 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.226377 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.226378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.226379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.226444 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.226881 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.226882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.226883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.227027 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:36.227069 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.227275 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.227359 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:43:36.227730 8777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.193951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.0.0.1:40472)
0928 04:43:36.194457 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:36.194466 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:36.194543 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:36.194714 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:36.194715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:36.195048 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:36.195158 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.195160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.196128 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.196129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.197151 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.197153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.197687 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.197688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.197753 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.223381 (+ 25628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:36.223387 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:36.223963 (+ 576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:36.226046 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.226047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.226047 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.226288 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.226529 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.226529 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.226530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.226795 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.227114 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:36.227115 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:36.227168 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 04:43:36.227529 (+ 361us) client_negotiation.cc:240] Negotiation successful
0928 04:43:36.227629 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":71,"thread_start_us":79,"threads_started":1}
Sep 28 04:43:36 dist-test-slave-j5c3 krb5kdc[7988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570615, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.254@KRBTEST.COM for oryx/127.1.226.252@KRBTEST.COM
I20260928 04:43:36.238499 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.230068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:45505 (local address 127.0.0.1:37778)
0928 04:43:36.230114 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:36.230124 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:36.230158 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:36.230287 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:36.230288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:36.230452 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:36.230501 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.230503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.231441 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.231442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.232300 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.232303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.232804 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.232805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.232877 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.233319 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:36.233324 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:36.235015 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:36.237165 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.237166 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.237166 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.237426 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.237664 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.237665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.237666 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.237753 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.238068 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:36.238069 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:36.238119 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 04:43:36.238207 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 04:43:36.238329 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 04:43:36.238895 8755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.230124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37778 (local address 127.1.226.252:45505)
0928 04:43:36.230172 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.230174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.230182 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:43:36.230217 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.230218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.230228 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.230285 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:36.230581 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.231350 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.232385 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.232713 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.232792 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.235120 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:36.235122 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:36.235122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:36.235130 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:36.237055 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.237529 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.237530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.237530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.237590 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.237829 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.237830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.237830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.237974 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:36.238039 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.238712 (+ 673us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.238794 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 04:43:36.239357 8032 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.226.252:45505: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956"
member_type: NON_VOTER
last_known_addr {
host: "127.1.226.252"
port: 45505
}
attrs {
promote: true
}
}
dest_uuid: "ab16fb1ab1aa488299d5049c1f16fa81"
cas_config_opid_index: 5
I20260928 04:43:36.239609 8032 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: NON_VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: true } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: NON_VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: true } }
I20260928 04:43:36.240008 8593 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2aec508a2bbf4925857d0d1a1d010956 (127.1.226.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: NON_VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: true } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: NON_VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: true } } }
I20260928 04:43:36.241074 8593 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.226.252:45505
I20260928 04:43:36.241145 8593 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: NON_VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: true } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: NON_VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: true } } }
I20260928 04:43:36.241209 8593 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:36.241637 8727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:36.242889 8727 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [term 1 FOLLOWER]: Refusing update from remote peer ab16fb1ab1aa488299d5049c1f16fa81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 04:43:36.243335 8685 master.cc:537] Master@127.1.226.252:45505 shutting down...
I20260928 04:43:36.243316 8592 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: NON_VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 04:43:36.244066 8020 consensus_peers.cc:487] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 -> Peer 2aec508a2bbf4925857d0d1a1d010956 (127.1.226.252:45505): Couldn't send request to peer 2aec508a2bbf4925857d0d1a1d010956. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 04:43:36.246448 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.240195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34024 (local address 127.1.226.253:40789)
0928 04:43:36.241288 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.241291 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.241301 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:43:36.241327 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.241328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.241352 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.241433 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:36.241600 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.242378 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.245422 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.246034 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.246114 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.246189 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.246330 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.246378 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1068}
I20260928 04:43:36.246663 8685 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:43:36.246753 8685 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:36.246800 8685 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956: stopping tablet replica
I20260928 04:43:36.247059 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.240121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:40789 (local address 127.0.0.1:34024)
0928 04:43:36.240299 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:36.240310 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:36.240347 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:36.241461 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:36.241462 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:36.241463 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:36.241506 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.241508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.242475 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.242475 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.245321 (+ 2846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.245326 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.246123 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.246124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.246205 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.246233 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 04:43:36.246945 (+ 712us) client_negotiation.cc:240] Negotiation successful
0928 04:43:36.246993 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260928 04:43:36.247334 8641 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [term 1 LEARNER]: Refusing update from remote peer ab16fb1ab1aa488299d5049c1f16fa81: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 04:43:36.247536 8592 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: NON_VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 04:43:36.248082 8641 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2aec508a2bbf4925857d0d1a1d010956 (127.1.226.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: NON_VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: true } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: NON_VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: true } } }
I20260928 04:43:36.249495 8670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab16fb1ab1aa488299d5049c1f16fa81. Latest consensus state: current_term: 1 leader_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: NON_VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: true } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: NON_VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: true } } }
I20260928 04:43:36.249583 8670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [sys.catalog]: This master's current role is: LEARNER
I20260928 04:43:36.249616 8670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: NON_VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: true } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: NON_VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: true } } }
I20260928 04:43:36.249665 8670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [sys.catalog]: This master's current role is: LEARNER
I20260928 04:43:36.249886 8592 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81: attempting to promote NON_VOTER 8030293b08fe463b9af2092f0c6a7233 to VOTER
I20260928 04:43:36.250102 8592 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: NON_VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: true } }
I20260928 04:43:36.250667 8641 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [term 1 LEARNER]: Refusing update from remote peer ab16fb1ab1aa488299d5049c1f16fa81: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 04:43:36.250886 8781 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260928 04:43:36.251267 8020 connection.cc:535] client connection to 127.1.226.252:45505 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 04:43:36.251317 8020 consensus_peers.cc:487] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 -> Peer 2aec508a2bbf4925857d0d1a1d010956 (127.1.226.252:45505): Couldn't send request to peer 2aec508a2bbf4925857d0d1a1d010956. Status: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20260928 04:43:36.252002 8592 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8030293b08fe463b9af2092f0c6a7233 (127.1.226.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: NON_VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: true } } }
I20260928 04:43:36.252207 8641 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8030293b08fe463b9af2092f0c6a7233 (127.1.226.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: NON_VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: true } } }
I20260928 04:43:36.252964 8592 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: NON_VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: true } } }
I20260928 04:43:36.253038 8592 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:36.253088 8592 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: NON_VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: true } } }
I20260928 04:43:36.253130 8592 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:36.253314 8780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: NON_VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: true } } }
I20260928 04:43:36.253391 8780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:43:36.262953 8685 master.cc:552] Master@127.1.226.252:45505 shutdown complete.
I20260928 04:43:36.263013 8685 master_runner.cc:305] Clearing existing system tablet
I20260928 04:43:36.263141 8685 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:36.263190 8685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:36.263967 8685 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:36.264616 8784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:36.264775 8685 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:36.264842 8685 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:36.264865 8685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-2/wal
uuid: "2aec508a2bbf4925857d0d1a1d010956"
format_stamp: "Formatted at 2026-09-28 04:43:36 on dist-test-slave-j5c3"
server_key: "4149979846eb323fd3e322a00445855e"
server_key_iv: "5dbc4fba9eb4abd749d10caf6a020579"
server_key_version: "encryptionkey@0"
I20260928 04:43:36.265614 8685 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:36.267158 8685 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 04:43:36.267226 8685 log.cc:1196] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 04:43:36.267452 8685 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956: Deleting consensus metadata
I20260928 04:43:36.267582 8685 master_runner.cc:315] Copying system tablet from 127.1.226.254:42709
I20260928 04:43:36.268221 8685 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956: tablet copy: Beginning tablet copy session from remote peer at address 127.1.226.254:42709
I20260928 04:43:36.275763 8073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.268711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40480 (local address 127.1.226.254:42709)
0928 04:43:36.268756 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.268758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.268769 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:43:36.268841 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.268841 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.268852 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.268919 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:36.269287 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.270079 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.271161 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.271458 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.271530 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.272643 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:36.272644 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:36.272645 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:36.272652 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:36.274378 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.274806 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.274807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.274808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.274871 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.275118 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.275119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.275119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.275254 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:36.275319 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.275530 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.275628 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:43:36.276398 8790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.268538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.0.0.1:40480)
0928 04:43:36.268683 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:36.268698 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:36.268770 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:36.268938 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:36.268939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:36.269145 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:36.269202 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.269204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.270173 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.270175 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.271063 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.271065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.271543 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.271544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.271603 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.272116 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:36.272121 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:36.272550 (+ 429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:36.274471 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.274472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.274473 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.274720 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.274945 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.274946 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.274947 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.275047 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.275370 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:36.275371 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:36.275417 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 04:43:36.276050 (+ 633us) client_negotiation.cc:240] Negotiation successful
0928 04:43:36.276234 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
I20260928 04:43:36.276830 8053 tablet_copy_service.cc:140] P ab16fb1ab1aa488299d5049c1f16fa81: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2aec508a2bbf4925857d0d1a1d010956 ({username='oryx', principal='oryx/127.1.226.252@KRBTEST.COM'} at 127.0.0.1:40480)
I20260928 04:43:36.276901 8053 tablet_copy_service.cc:161] P ab16fb1ab1aa488299d5049c1f16fa81: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2aec508a2bbf4925857d0d1a1d010956 at {username='oryx', principal='oryx/127.1.226.252@KRBTEST.COM'} at 127.0.0.1:40480: session id = 2aec508a2bbf4925857d0d1a1d010956-00000000000000000000000000000000
I20260928 04:43:36.277505 8053 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 04:43:36.278002 8685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 04:43:36.279345 8685 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956: tablet copy: Starting download of 0 data blocks...
I20260928 04:43:36.279502 8685 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956: tablet copy: Starting download of 1 WAL segments...
I20260928 04:43:36.280309 8685 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 04:43:36.281286 8053 tablet_copy_service.cc:342] P ab16fb1ab1aa488299d5049c1f16fa81: Request end of tablet copy session 2aec508a2bbf4925857d0d1a1d010956-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.226.252@KRBTEST.COM'} at 127.0.0.1:40480
I20260928 04:43:36.281342 8053 tablet_copy_service.cc:433] P ab16fb1ab1aa488299d5049c1f16fa81: ending tablet copy session 2aec508a2bbf4925857d0d1a1d010956-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2aec508a2bbf4925857d0d1a1d010956
I20260928 04:43:36.284075 8685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:36.285375 8794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:36.285588 8791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:36.285658 8792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:36.285698 8685 server_base.cc:1034] running on GCE node
I20260928 04:43:36.285909 8685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:36.286056 8685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:36.287207 8685 hybrid_clock.cc:648] HybridClock initialized: now 1790570616287186 us; error 39 us; skew 500 ppm
Sep 28 04:43:36 dist-test-slave-j5c3 krb5kdc[7988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570616, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:36.288779 8685 init.cc:377] Logged in from keytab as oryx/127.1.226.252@KRBTEST.COM (short username oryx)
I20260928 04:43:36.289067 8685 webserver.cc:460] Webserver started at http://127.1.226.252:45005/ using document root <none> and password file <none>
I20260928 04:43:36.289191 8685 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:36.289232 8685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:36.290042 8685 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:36.290657 8801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:36.290836 8685 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:36.290894 8685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-2/wal
uuid: "2aec508a2bbf4925857d0d1a1d010956"
format_stamp: "Formatted at 2026-09-28 04:43:36 on dist-test-slave-j5c3"
server_key: "4149979846eb323fd3e322a00445855e"
server_key_iv: "5dbc4fba9eb4abd749d10caf6a020579"
server_key_version: "encryptionkey@0"
I20260928 04:43:36.290977 8685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:36.323194 8685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:36.325237 8685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:36.328476 8685 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.252:45505
I20260928 04:43:36.328583 8853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.252:45505 every 8 connection(s)
I20260928 04:43:36.328828 8685 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/master-2/data/info.pb
I20260928 04:43:36.329815 8857 sys_catalog.cc:263] Verifying existing consensus state
I20260928 04:43:36.330297 8857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956: Bootstrap starting.
W20260928 04:43:36.330947 8778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:36.331156 8778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:36.331198 8778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:36.332628 8778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:36.332697 8778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:36.332746 8778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:43:36.333261 8857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:43:36.333474 8857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956: Bootstrap complete.
I20260928 04:43:36.333657 8857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: NON_VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: true } }
I20260928 04:43:36.333748 8857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2aec508a2bbf4925857d0d1a1d010956, State: Initialized, Role: LEARNER
I20260928 04:43:36.333793 8857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: NON_VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: true } }
I20260928 04:43:36.333962 8857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:43:36.333972 8860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: NON_VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: true } } }
I20260928 04:43:36.334187 8860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [sys.catalog]: This master's current role is: LEARNER
I20260928 04:43:36.334719 8778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.193.keytab
--principal=oryx/127.1.226.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:43207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=40891
--tserver_master_addrs=127.1.226.254:42709,127.1.226.253:40789,127.1.226.252:45505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:36.334975 8778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:36.335338 8778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:43:36.335522 8871 catalog_manager.cc:1259] Loaded cluster ID: 2e45c915021c4b2b9faa0da3abf6c355
I20260928 04:43:36.335605 8871 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956: loading cluster ID for follower catalog manager: success
I20260928 04:43:36.336925 8871 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956: acquiring CA information for follower catalog manager: success
I20260928 04:43:36.337272 8871 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 04:43:36.341795 8876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:36.341845 8778 server_base.cc:1034] running on GCE node
W20260928 04:43:36.342037 8874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:36.341809 8873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:36.342363 8778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:36.342633 8778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:36.343812 8778 hybrid_clock.cc:648] HybridClock initialized: now 1790570616343786 us; error 35 us; skew 500 ppm
Sep 28 04:43:36 dist-test-slave-j5c3 krb5kdc[7988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570616, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:36.346676 8778 init.cc:377] Logged in from keytab as oryx/127.1.226.193@KRBTEST.COM (short username oryx)
I20260928 04:43:36.347163 8778 webserver.cc:460] Webserver started at http://127.1.226.193:40891/ using document root <none> and password file <none>
I20260928 04:43:36.347405 8778 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:36.347460 8778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:36.349069 8778 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:43:36.349776 8883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:36.349957 8778 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:43:36.350056 8778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "554061b51b954cdf83cd7da7df877b51"
format_stamp: "Formatted at 2026-09-28 04:43:35 on dist-test-slave-j5c3"
server_key: "d94b070a58d81228fbc17d250ab38b62"
server_key_iv: "472cce503a4c9f63a5906d5bb969cc3d"
server_key_version: "encryptionkey@0"
I20260928 04:43:36.350423 8778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:36.362018 8778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:36.364472 8778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:36.364727 8778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:36.364956 8778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:36.365265 8778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:36.365298 8778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:36.365320 8778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:36.365334 8778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:36.371245 8778 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:43207
I20260928 04:43:36.371352 8996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:43207 every 8 connection(s)
I20260928 04:43:36.371685 8778 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
I20260928 04:43:36.376178 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 8778
I20260928 04:43:36.376358 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 8229
Sep 28 04:43:36 dist-test-slave-j5c3 krb5kdc[7988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570616, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.252@KRBTEST.COM
Sep 28 04:43:36 dist-test-slave-j5c3 krb5kdc[7988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570616, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM
I20260928 04:43:36.381983 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:33589
--local_ip_for_outbound_sockets=127.1.226.194
--tserver_master_addrs=127.1.226.254:42709
--webserver_port=35161
--webserver_interface=127.1.226.194
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.194.keytab
--principal=oryx/127.1.226.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.226.254:42709,127.1.226.253:40789
--tserver_master_addrs=127.1.226.254:42709,127.1.226.253:40789,127.1.226.252:45505 with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 04:43:36 dist-test-slave-j5c3 krb5kdc[7988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570616, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.253@KRBTEST.COM
I20260928 04:43:36.385088 9001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.372447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:45505 (local address 127.1.226.193:38981)
0928 04:43:36.372621 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:36.372664 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:36.373742 (+ 1078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:36.374029 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:36.374030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:36.374412 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:36.374644 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.374649 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.375815 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.375816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.377468 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.377471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.377943 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.377944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.378072 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.378628 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:36.378637 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:36.380725 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:36.383173 (+ 2448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.383180 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.383182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.383591 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.383833 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.383834 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.383834 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.383969 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.384306 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:36.384314 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:36.384482 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 04:43:36.384595 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 04:43:36.384815 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":70,"thread_start_us":111,"threads_started":1}
I20260928 04:43:36.385160 8073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.372476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:34051 (local address 127.1.226.254:42709)
0928 04:43:36.372548 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.372551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.372742 (+ 191us) server_negotiation.cc:407] Connection header received
0928 04:43:36.373864 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.373865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.373879 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.373929 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:36.374862 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.375464 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.378227 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.378514 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.378554 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.380841 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:36.380843 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:36.380843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:36.380851 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:36.383266 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.383768 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.383770 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.383770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.383825 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.384058 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.384059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.384060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.384486 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:36.384534 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.384966 (+ 432us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.385049 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 04:43:36.385514 9002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.372447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.1.226.193:34051)
0928 04:43:36.372622 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:36.372665 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:36.373745 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:36.374014 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:36.374017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:36.374395 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:36.374645 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.374650 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.375570 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.375571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.378125 (+ 2554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.378128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.378626 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.378627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.378686 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.379111 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:36.379116 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:36.380725 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:36.383397 (+ 2672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.383398 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.383398 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.383683 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.383898 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.383899 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.383899 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.383969 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.384585 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:36.384587 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:36.384621 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 04:43:36.385233 (+ 612us) client_negotiation.cc:240] Negotiation successful
0928 04:43:36.385349 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":117,"thread_start_us":95,"threads_started":1}
I20260928 04:43:36.386617 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.372902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:35771 (local address 127.1.226.253:40789)
0928 04:43:36.372947 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.372949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.372959 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:43:36.373872 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.373874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.373887 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.373919 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:36.374748 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.375541 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.381209 (+ 5668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.381578 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.381665 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.383932 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:36.383933 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:36.383933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:36.383942 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:36.385489 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.385893 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.385894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.385894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.385948 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.386120 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.386120 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.386121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.386244 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:36.386281 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.386455 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.386528 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 04:43:36.386965 9003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.372752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:40789 (local address 127.1.226.193:35771)
0928 04:43:36.372889 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:36.372898 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:36.373753 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:36.374014 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:36.374017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:36.374422 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:36.374644 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.374649 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.375640 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.375686 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.381126 (+ 5440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.381129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.381670 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.381671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.381732 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.382244 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:36.382249 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:36.383827 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:36.385614 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.385615 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.385615 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.385819 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.386013 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.386013 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.386014 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.386062 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.386323 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:36.386323 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:36.386360 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 04:43:36.386703 (+ 343us) client_negotiation.cc:240] Negotiation successful
0928 04:43:36.386821 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":111,"thread_start_us":64,"threads_started":1}
I20260928 04:43:36.387311 8998 heartbeater.cc:347] Connected to a master server at 127.1.226.254:42709
I20260928 04:43:36.387432 8998 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:36.387614 8998 heartbeater.cc:511] Master 127.1.226.254:42709 requested a full tablet report, sending...
I20260928 04:43:36.388067 8032 ts_manager.cc:194] Re-registered known tserver with Master: 554061b51b954cdf83cd7da7df877b51 (127.1.226.193:43207)
I20260928 04:43:36.388680 8032 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.226.193@KRBTEST.COM'} at 127.1.226.193:34051
I20260928 04:43:36.388859 8999 heartbeater.cc:347] Connected to a master server at 127.1.226.253:40789
I20260928 04:43:36.388912 8999 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:36.388984 8999 heartbeater.cc:511] Master 127.1.226.253:40789 requested a full tablet report, sending...
I20260928 04:43:36.389421 8631 ts_manager.cc:194] Registered new tserver with Master: 554061b51b954cdf83cd7da7df877b51 (127.1.226.193:43207)
I20260928 04:43:36.396467 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.372766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:38981 (local address 127.1.226.252:45505)
0928 04:43:36.372952 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.372954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.372964 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:43:36.373910 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.373911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.373924 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.373987 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:36.374749 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.375736 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.377570 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.377796 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.377935 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.380847 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:36.380848 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:36.380848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:36.380856 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:36.383068 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.383697 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.383698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.383699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.383764 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.384039 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.384039 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.384040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.384215 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:36.384279 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.396283 (+ 12004us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.396391 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":55,"threads_started":1}
I20260928 04:43:36.396945 8997 heartbeater.cc:347] Connected to a master server at 127.1.226.252:45505
I20260928 04:43:36.397024 8997 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:36.397120 8997 heartbeater.cc:511] Master 127.1.226.252:45505 requested a full tablet report, sending...
I20260928 04:43:36.397503 8818 ts_manager.cc:194] Registered new tserver with Master: 554061b51b954cdf83cd7da7df877b51 (127.1.226.193:43207)
W20260928 04:43:36.470678 9005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:36.470938 9005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:36.470966 9005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:36.472146 9005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:36.472209 9005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:36.472265 9005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:43:36.473778 9005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.194.keytab
--principal=oryx/127.1.226.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:33589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=35161
--tserver_master_addrs=127.1.226.254:42709,127.1.226.253:40789,127.1.226.252:45505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:36.474042 9005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:36.474327 9005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:36.479854 9011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:36.479851 9010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:36.479862 9013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:36.480239 9005 server_base.cc:1034] running on GCE node
I20260928 04:43:36.480464 9005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:36.480747 9005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:36.481868 9005 hybrid_clock.cc:648] HybridClock initialized: now 1790570616481852 us; error 24 us; skew 500 ppm
Sep 28 04:43:36 dist-test-slave-j5c3 krb5kdc[7988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570616, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:36.484875 9005 init.cc:377] Logged in from keytab as oryx/127.1.226.194@KRBTEST.COM (short username oryx)
I20260928 04:43:36.485394 9005 webserver.cc:460] Webserver started at http://127.1.226.194:35161/ using document root <none> and password file <none>
I20260928 04:43:36.485675 9005 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:36.485770 9005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:36.487315 9005 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:36.487928 9020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:36.488129 9005 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:43:36.488255 9005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "fc4ea0c4703a4ab998401c9357d616da"
format_stamp: "Formatted at 2026-09-28 04:43:35 on dist-test-slave-j5c3"
server_key: "6a8bf799b5f3011e8694f5a2c29e6717"
server_key_iv: "3293070157c141390f6e6fc303940861"
server_key_version: "encryptionkey@0"
I20260928 04:43:36.488631 9005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:36.500433 9005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:36.502942 9005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:36.503093 9005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:36.503358 9005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:36.503686 9005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:36.503723 9005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:36.503746 9005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:36.503791 9005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:36.510244 9005 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:33589
I20260928 04:43:36.510308 9133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:33589 every 8 connection(s)
I20260928 04:43:36.510712 9005 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
Sep 28 04:43:36 dist-test-slave-j5c3 krb5kdc[7988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570616, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.252@KRBTEST.COM
Sep 28 04:43:36 dist-test-slave-j5c3 krb5kdc[7988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570616, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM
I20260928 04:43:36.518540 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 9005
I20260928 04:43:36.518712 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 8363
Sep 28 04:43:36 dist-test-slave-j5c3 krb5kdc[7988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570616, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.253@KRBTEST.COM
I20260928 04:43:36.521955 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.511582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:36753 (local address 127.1.226.252:45505)
0928 04:43:36.511627 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.511630 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.511795 (+ 165us) server_negotiation.cc:407] Connection header received
0928 04:43:36.512874 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.512875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.512892 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.512949 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:36.513703 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.514197 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.515207 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.515432 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.515521 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.518222 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:36.518223 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:36.518223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:36.518230 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:36.520027 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.520603 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.520604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.520605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.520668 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.520999 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.521000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.521000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.521168 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:36.521237 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.521752 (+ 515us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.521846 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:43:36.522522 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.511476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:45505 (local address 127.1.226.194:36753)
0928 04:43:36.511642 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:36.511686 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:36.512755 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:36.513026 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:36.513029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:36.513381 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:36.513597 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.513601 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.514301 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.514302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.515096 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.515098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.515531 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.515532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.515660 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.516424 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:36.516433 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:36.518119 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:36.520118 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.520125 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.520127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.520523 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.520752 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.520753 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.520753 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.520889 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.521370 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:36.521377 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:36.521548 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 04:43:36.522053 (+ 505us) client_negotiation.cc:240] Negotiation successful
0928 04:43:36.522253 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":2,"thread_start_us":96,"threads_started":1}
I20260928 04:43:36.523110 8073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.511471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:37179 (local address 127.1.226.254:42709)
0928 04:43:36.511551 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.511553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.511765 (+ 212us) server_negotiation.cc:407] Connection header received
0928 04:43:36.512897 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.512898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.512911 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.512942 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:36.513704 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.514454 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.515563 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.515868 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.515904 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.518736 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:36.518738 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:36.518738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:36.518747 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:36.521186 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.521864 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.521866 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.521866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.521926 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.522401 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.522402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.522403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.522547 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:36.522591 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.522930 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.523022 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 04:43:36.523370 9139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.511478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.1.226.194:37179)
0928 04:43:36.511700 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:36.511711 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:36.512781 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:36.513025 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:36.513029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:36.513410 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:36.513597 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.513603 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.514556 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.514557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.515463 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.515466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.515979 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.515980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.516046 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.516884 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:36.516888 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:36.518619 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:36.521271 (+ 2652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.521272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.521273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.521776 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.522229 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.522230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.522230 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.522311 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.522764 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:36.522765 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:36.522808 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 04:43:36.522876 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 04:43:36.522987 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":106,"thread_start_us":82,"threads_started":1}
I20260928 04:43:36.523556 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.511879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:35053 (local address 127.1.226.253:40789)
0928 04:43:36.511925 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.511927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.511975 (+ 48us) server_negotiation.cc:407] Connection header received
0928 04:43:36.512885 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.512887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.512900 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.512933 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:36.513703 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.514468 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.515563 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.515857 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.515899 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.520609 (+ 4710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:36.520610 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:36.520610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:36.520618 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:36.522311 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.522717 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.522718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.522718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.522783 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.523021 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.523022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.523022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.523158 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:36.523204 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.523400 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.523472 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 04:43:36.523782 9140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.511773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:40789 (local address 127.1.226.194:35053)
0928 04:43:36.511923 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:36.511932 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:36.512781 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:36.513031 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:36.513032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:36.513409 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:36.513597 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.513603 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.514556 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.514557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.515469 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.515470 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.515980 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.515981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.516046 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.517072 (+ 1026us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:36.517077 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:36.520502 (+ 3425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:36.522408 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.522409 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.522409 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.522630 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.522884 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.522885 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.522885 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.522945 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.523259 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:36.523260 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:36.523291 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 04:43:36.523363 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 04:43:36.523465 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":63,"thread_start_us":98,"threads_started":1}
I20260928 04:43:36.524593 9135 heartbeater.cc:347] Connected to a master server at 127.1.226.254:42709
I20260928 04:43:36.524617 9136 heartbeater.cc:347] Connected to a master server at 127.1.226.253:40789
I20260928 04:43:36.524725 9136 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:36.524724 9135 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:36.524928 9136 heartbeater.cc:511] Master 127.1.226.253:40789 requested a full tablet report, sending...
I20260928 04:43:36.524930 9135 heartbeater.cc:511] Master 127.1.226.254:42709 requested a full tablet report, sending...
I20260928 04:43:36.525449 8631 ts_manager.cc:194] Registered new tserver with Master: fc4ea0c4703a4ab998401c9357d616da (127.1.226.194:33589)
I20260928 04:43:36.525454 8032 ts_manager.cc:194] Re-registered known tserver with Master: fc4ea0c4703a4ab998401c9357d616da (127.1.226.194:33589)
I20260928 04:43:36.525954 8032 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.226.194@KRBTEST.COM'} at 127.1.226.194:37179
I20260928 04:43:36.525981 9134 heartbeater.cc:347] Connected to a master server at 127.1.226.252:45505
I20260928 04:43:36.526028 9134 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:36.526103 9134 heartbeater.cc:511] Master 127.1.226.252:45505 requested a full tablet report, sending...
I20260928 04:43:36.526142 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:45509
--local_ip_for_outbound_sockets=127.1.226.195
--tserver_master_addrs=127.1.226.254:42709
--webserver_port=44397
--webserver_interface=127.1.226.195
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.195.keytab
--principal=oryx/127.1.226.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.226.254:42709,127.1.226.253:40789
--tserver_master_addrs=127.1.226.254:42709,127.1.226.253:40789,127.1.226.252:45505 with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:43:36.526397 8818 ts_manager.cc:194] Registered new tserver with Master: fc4ea0c4703a4ab998401c9357d616da (127.1.226.194:33589)
W20260928 04:43:36.612278 9141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:36.612507 9141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:36.612533 9141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:36.613669 9141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:36.613711 9141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:36.613739 9141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:43:36.615262 9141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.195.keytab
--principal=oryx/127.1.226.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:45509
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=44397
--tserver_master_addrs=127.1.226.254:42709,127.1.226.253:40789,127.1.226.252:45505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:36.615470 9141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:36.615674 9141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:36.620967 9147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:36.621029 9149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:36.620963 9146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:36.621001 9141 server_base.cc:1034] running on GCE node
I20260928 04:43:36.621383 9141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:36.621675 9141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:36.622848 9141 hybrid_clock.cc:648] HybridClock initialized: now 1790570616622818 us; error 41 us; skew 500 ppm
Sep 28 04:43:36 dist-test-slave-j5c3 krb5kdc[7988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570616, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:36.625613 9141 init.cc:377] Logged in from keytab as oryx/127.1.226.195@KRBTEST.COM (short username oryx)
I20260928 04:43:36.626106 9141 webserver.cc:460] Webserver started at http://127.1.226.195:44397/ using document root <none> and password file <none>
I20260928 04:43:36.626380 9141 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:36.626478 9141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:36.627981 9141 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:43:36.628715 9156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:36.628914 9141 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:43:36.629030 9141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "033aee8210cc40bf922b90e5199137c4"
format_stamp: "Formatted at 2026-09-28 04:43:35 on dist-test-slave-j5c3"
server_key: "129ff101f8d91300ff0cce2a311e3fdc"
server_key_iv: "e99204d91a432189a2f5187c4e75af47"
server_key_version: "encryptionkey@0"
I20260928 04:43:36.629417 9141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:36.662475 9141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:36.665041 9141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:36.665202 9141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:36.665450 9141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:36.665781 9141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:36.665830 9141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:36.665884 9141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:36.665907 9141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:36.671885 9141 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:45509
I20260928 04:43:36.671957 9269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:45509 every 8 connection(s)
I20260928 04:43:36.672384 9141 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
Sep 28 04:43:36 dist-test-slave-j5c3 krb5kdc[7988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570616, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.253@KRBTEST.COM
Sep 28 04:43:36 dist-test-slave-j5c3 krb5kdc[7988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570616, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM
Sep 28 04:43:36 dist-test-slave-j5c3 krb5kdc[7988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570616, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.252@KRBTEST.COM
I20260928 04:43:36.681572 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 9141
I20260928 04:43:36.684585 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.673150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:48105 (local address 127.1.226.253:40789)
0928 04:43:36.673273 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.673275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.673533 (+ 258us) server_negotiation.cc:407] Connection header received
0928 04:43:36.674858 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.674859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.674870 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.674911 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:36.675813 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.676605 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.677788 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.678081 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.678131 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.680568 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:36.680570 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:36.680570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:36.680578 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:36.682708 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.683190 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.683192 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.683192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.683292 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.683793 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.683794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.683795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.683937 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:36.683982 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.684413 (+ 431us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.684492 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260928 04:43:36.684623 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.673614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:60365 (local address 127.1.226.252:45505)
0928 04:43:36.673663 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.673665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.673814 (+ 149us) server_negotiation.cc:407] Connection header received
0928 04:43:36.674962 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.674964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.674971 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.675034 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:36.675799 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.676513 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.677636 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.677934 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.678000 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.681547 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:36.681549 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:36.681549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:36.681557 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:36.683449 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.683833 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.683834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.683835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.683882 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.684080 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.684081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.684081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.684221 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:36.684264 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.684464 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.684535 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 04:43:36.685276 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.673154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:40789 (local address 127.1.226.195:48105)
0928 04:43:36.673394 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:36.673438 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:36.674662 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:36.674980 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:36.674984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:36.675449 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:36.675691 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.675697 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.676707 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.676708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.677698 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.677700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.678147 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.678147 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.678198 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.678835 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:36.678846 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:36.680476 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:36.682790 (+ 2314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.682797 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.682799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.683092 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.683549 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.683549 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.683550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.683705 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.684028 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:36.684033 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:36.684229 (+ 196us) client_negotiation.cc:769] Sending connection context
0928 04:43:36.684676 (+ 447us) client_negotiation.cc:240] Negotiation successful
0928 04:43:36.685011 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":40,"thread_start_us":85,"threads_started":1}
I20260928 04:43:36.685273 9274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.673496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:45505 (local address 127.1.226.195:60365)
0928 04:43:36.673771 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:36.673779 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:36.674673 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:36.675055 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:36.675056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:36.675446 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:36.675690 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.675695 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.676620 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.676621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.677543 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.677545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.678012 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.678013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.678135 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.678953 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:36.678959 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:36.681441 (+ 2482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:36.683532 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.683533 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.683534 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.683742 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.683952 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.683953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.683954 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.684010 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.684313 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:36.684314 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:36.684351 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 04:43:36.684695 (+ 344us) client_negotiation.cc:240] Negotiation successful
0928 04:43:36.685062 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":158,"thread_start_us":57,"threads_started":1}
I20260928 04:43:36.685885 9272 heartbeater.cc:347] Connected to a master server at 127.1.226.253:40789
I20260928 04:43:36.685994 9272 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:36.686122 9270 heartbeater.cc:347] Connected to a master server at 127.1.226.252:45505
I20260928 04:43:36.686157 9270 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:36.686156 8073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.673215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:48519 (local address 127.1.226.254:42709)
0928 04:43:36.673289 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.673292 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.673733 (+ 441us) server_negotiation.cc:407] Connection header received
0928 04:43:36.675093 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.675095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.675106 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.675165 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:36.675806 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.676634 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.678178 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.678404 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.678470 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.681021 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:36.681022 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:36.681022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:36.681032 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:36.683688 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.684121 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.684123 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.684124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.684212 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.684461 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.684462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.684462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.684595 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:36.684666 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.685973 (+ 1307us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.686062 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 04:43:36.686209 9272 heartbeater.cc:511] Master 127.1.226.253:40789 requested a full tablet report, sending...
I20260928 04:43:36.686209 9270 heartbeater.cc:511] Master 127.1.226.252:45505 requested a full tablet report, sending...
I20260928 04:43:36.686710 8818 ts_manager.cc:194] Registered new tserver with Master: 033aee8210cc40bf922b90e5199137c4 (127.1.226.195:45509)
I20260928 04:43:36.686738 8631 ts_manager.cc:194] Registered new tserver with Master: 033aee8210cc40bf922b90e5199137c4 (127.1.226.195:45509)
I20260928 04:43:36.687173 9276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.673155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.1.226.195:48519)
0928 04:43:36.673679 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:36.673689 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:36.674662 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:36.675181 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:36.675182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:36.675515 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:36.675691 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.675697 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.676741 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.676742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.677702 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.677703 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.678495 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.678496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.678548 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.679038 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:36.679044 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:36.680923 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:36.683781 (+ 2858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.683782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.683782 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.684022 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.684306 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:36.684307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:36.684307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:36.684378 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:36.685259 (+ 881us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:36.685260 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:36.685304 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 04:43:36.686498 (+ 1194us) client_negotiation.cc:240] Negotiation successful
0928 04:43:36.687042 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":107,"thread_start_us":167,"threads_started":1}
I20260928 04:43:36.687526 9271 heartbeater.cc:347] Connected to a master server at 127.1.226.254:42709
I20260928 04:43:36.687582 9271 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:36.687656 9271 heartbeater.cc:511] Master 127.1.226.254:42709 requested a full tablet report, sending...
I20260928 04:43:36.687937 8032 ts_manager.cc:194] Re-registered known tserver with Master: 033aee8210cc40bf922b90e5199137c4 (127.1.226.195:45509)
I20260928 04:43:36.688457 8032 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.226.195@KRBTEST.COM'} at 127.1.226.195:48519
I20260928 04:43:36.692344 9281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.683038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40494 (local address 127.1.226.254:42709)
0928 04:43:36.683454 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.683457 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.683653 (+ 196us) server_negotiation.cc:407] Connection header received
0928 04:43:36.683676 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.683677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.683691 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.683757 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:36.684728 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.685940 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.687057 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.687430 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.687498 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.688977 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:36.688978 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:36.688979 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:36.688986 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:36.690940 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.691341 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.691342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.691343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.691445 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.691690 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.691691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.691691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.691825 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:36.691932 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.692112 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.692225 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":99,"threads_started":1}
I20260928 04:43:36.695196 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.687737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34032 (local address 127.1.226.253:40789)
0928 04:43:36.687780 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.687782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.687978 (+ 196us) server_negotiation.cc:407] Connection header received
0928 04:43:36.688044 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.688045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.688056 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.688120 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:36.688918 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.689771 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.690499 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.690779 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.690838 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.692382 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:36.692383 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:36.692383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:36.692390 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:36.694070 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.694434 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.694435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.694435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.694483 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.694661 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.694663 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.694663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.694797 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:36.694879 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.695037 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.695108 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 04:43:36.694008 8032 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:43:36.696307 8032 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:43:36.696864 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.688182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37784 (local address 127.1.226.252:45505)
0928 04:43:36.688224 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.688226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.689057 (+ 831us) server_negotiation.cc:407] Connection header received
0928 04:43:36.689157 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.689158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.689168 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.689195 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:36.689795 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.690560 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.691733 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.692033 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.692108 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.693695 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:36.693696 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:36.693697 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:36.693704 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:36.695492 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.695929 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.695931 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.695931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.695990 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.696265 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.696266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.696266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.696408 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:36.696489 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.696683 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.696761 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 04:43:36.704020 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.700208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:33589 (local address 127.0.0.1:44874)
0928 04:43:36.700499 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:36.700510 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:36.700563 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:36.700953 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:36.700954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:36.700957 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:36.701006 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.701008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.701871 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.701873 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.702936 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.702938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.703775 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.703776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.703833 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.703847 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 04:43:36.703885 (+ 38us) client_negotiation.cc:240] Negotiation successful
0928 04:43:36.703926 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":87,"threads_started":1}
I20260928 04:43:36.704016 9293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.700606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44874 (local address 127.1.226.194:33589)
0928 04:43:36.700792 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.700795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.700804 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:43:36.700835 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.700836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.700849 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.700938 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:36.701063 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.701770 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.703029 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.703645 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.703773 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.703863 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.703907 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.703942 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
I20260928 04:43:36.704532 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.700104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:45509 (local address 127.0.0.1:47944)
0928 04:43:36.700447 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:36.700457 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:36.700509 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:36.700807 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:36.700808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:36.700810 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:36.700879 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.700881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.702204 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.702205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.703242 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.703245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.704275 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.704276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.704347 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.704351 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:43:36.704405 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 04:43:36.704450 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":23,"thread_start_us":143,"threads_started":1}
I20260928 04:43:36.704744 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.700088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:43207 (local address 127.0.0.1:51086)
0928 04:43:36.700214 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:36.700227 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:36.700273 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:36.700970 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:36.700971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:36.700973 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:36.701018 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.701019 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.702247 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.702248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.703261 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.703263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.704521 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.704521 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.704579 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.704582 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:43:36.704642 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 04:43:36.704680 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":87}
I20260928 04:43:36.704866 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.700186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47944 (local address 127.1.226.195:45509)
0928 04:43:36.700378 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.700383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.700565 (+ 182us) server_negotiation.cc:407] Connection header received
0928 04:43:36.700610 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.700612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.700646 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.700724 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:36.700968 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.702109 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.703335 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.704054 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.704123 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.704231 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.704723 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.704768 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":106,"threads_started":1}
I20260928 04:43:36.704820 9068 tablet_service.cc:1467] Processing CreateTablet for tablet 618be3997d304b0fa4f2a4f77867bb19 (DEFAULT_TABLE table=test-table [id=47b413bc5b6345239739568c796277af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:36.705220 9068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 618be3997d304b0fa4f2a4f77867bb19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:36.705471 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.700423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51086 (local address 127.1.226.193:43207)
0928 04:43:36.700805 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.700809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.700818 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:43:36.700856 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.700858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.700873 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.701088 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:36.701153 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.702162 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.703375 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.704042 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.704506 (+ 464us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.705327 (+ 821us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.705359 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.705411 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":78,"threads_started":1}
I20260928 04:43:36.705502 9204 tablet_service.cc:1467] Processing CreateTablet for tablet 618be3997d304b0fa4f2a4f77867bb19 (DEFAULT_TABLE table=test-table [id=47b413bc5b6345239739568c796277af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:36.705847 9204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 618be3997d304b0fa4f2a4f77867bb19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:36.708810 8931 tablet_service.cc:1467] Processing CreateTablet for tablet 618be3997d304b0fa4f2a4f77867bb19 (DEFAULT_TABLE table=test-table [id=47b413bc5b6345239739568c796277af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:36.709601 8931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 618be3997d304b0fa4f2a4f77867bb19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:36.709618 9294 tablet_bootstrap.cc:492] T 618be3997d304b0fa4f2a4f77867bb19 P fc4ea0c4703a4ab998401c9357d616da: Bootstrap starting.
I20260928 04:43:36.709725 9295 tablet_bootstrap.cc:492] T 618be3997d304b0fa4f2a4f77867bb19 P 033aee8210cc40bf922b90e5199137c4: Bootstrap starting.
I20260928 04:43:36.710328 9295 tablet_bootstrap.cc:654] T 618be3997d304b0fa4f2a4f77867bb19 P 033aee8210cc40bf922b90e5199137c4: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:36.710388 9294 tablet_bootstrap.cc:654] T 618be3997d304b0fa4f2a4f77867bb19 P fc4ea0c4703a4ab998401c9357d616da: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:36.710690 9295 log.cc:824] T 618be3997d304b0fa4f2a4f77867bb19 P 033aee8210cc40bf922b90e5199137c4: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:36.710708 9294 log.cc:824] T 618be3997d304b0fa4f2a4f77867bb19 P fc4ea0c4703a4ab998401c9357d616da: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:36.711447 9295 tablet_bootstrap.cc:492] T 618be3997d304b0fa4f2a4f77867bb19 P 033aee8210cc40bf922b90e5199137c4: No bootstrap required, opened a new log
I20260928 04:43:36.711638 9295 ts_tablet_manager.cc:1397] T 618be3997d304b0fa4f2a4f77867bb19 P 033aee8210cc40bf922b90e5199137c4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:43:36.711735 9294 tablet_bootstrap.cc:492] T 618be3997d304b0fa4f2a4f77867bb19 P fc4ea0c4703a4ab998401c9357d616da: No bootstrap required, opened a new log
I20260928 04:43:36.711802 9294 ts_tablet_manager.cc:1397] T 618be3997d304b0fa4f2a4f77867bb19 P fc4ea0c4703a4ab998401c9357d616da: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:43:36.713263 9298 tablet_bootstrap.cc:492] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51: Bootstrap starting.
I20260928 04:43:36.713910 9298 tablet_bootstrap.cc:654] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:36.713801 9295 raft_consensus.cc:348] T 618be3997d304b0fa4f2a4f77867bb19 P 033aee8210cc40bf922b90e5199137c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554061b51b954cdf83cd7da7df877b51" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43207 } } peers { permanent_uuid: "fc4ea0c4703a4ab998401c9357d616da" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 33589 } } peers { permanent_uuid: "033aee8210cc40bf922b90e5199137c4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45509 } }
I20260928 04:43:36.713961 9295 raft_consensus.cc:374] T 618be3997d304b0fa4f2a4f77867bb19 P 033aee8210cc40bf922b90e5199137c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:36.713996 9295 raft_consensus.cc:729] T 618be3997d304b0fa4f2a4f77867bb19 P 033aee8210cc40bf922b90e5199137c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 033aee8210cc40bf922b90e5199137c4, State: Initialized, Role: FOLLOWER
I20260928 04:43:36.714140 9295 consensus_queue.cc:260] T 618be3997d304b0fa4f2a4f77867bb19 P 033aee8210cc40bf922b90e5199137c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554061b51b954cdf83cd7da7df877b51" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43207 } } peers { permanent_uuid: "fc4ea0c4703a4ab998401c9357d616da" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 33589 } } peers { permanent_uuid: "033aee8210cc40bf922b90e5199137c4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45509 } }
I20260928 04:43:36.714121 9294 raft_consensus.cc:348] T 618be3997d304b0fa4f2a4f77867bb19 P fc4ea0c4703a4ab998401c9357d616da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554061b51b954cdf83cd7da7df877b51" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43207 } } peers { permanent_uuid: "fc4ea0c4703a4ab998401c9357d616da" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 33589 } } peers { permanent_uuid: "033aee8210cc40bf922b90e5199137c4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45509 } }
I20260928 04:43:36.714237 9298 log.cc:824] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:36.714255 9294 raft_consensus.cc:374] T 618be3997d304b0fa4f2a4f77867bb19 P fc4ea0c4703a4ab998401c9357d616da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:36.714296 9294 raft_consensus.cc:729] T 618be3997d304b0fa4f2a4f77867bb19 P fc4ea0c4703a4ab998401c9357d616da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc4ea0c4703a4ab998401c9357d616da, State: Initialized, Role: FOLLOWER
I20260928 04:43:36.714424 9295 ts_tablet_manager.cc:1428] T 618be3997d304b0fa4f2a4f77867bb19 P 033aee8210cc40bf922b90e5199137c4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:43:36.714394 9294 consensus_queue.cc:260] T 618be3997d304b0fa4f2a4f77867bb19 P fc4ea0c4703a4ab998401c9357d616da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554061b51b954cdf83cd7da7df877b51" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43207 } } peers { permanent_uuid: "fc4ea0c4703a4ab998401c9357d616da" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 33589 } } peers { permanent_uuid: "033aee8210cc40bf922b90e5199137c4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45509 } }
I20260928 04:43:36.714687 9294 ts_tablet_manager.cc:1428] T 618be3997d304b0fa4f2a4f77867bb19 P fc4ea0c4703a4ab998401c9357d616da: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:43:36.714840 9135 heartbeater.cc:503] Master 127.1.226.254:42709 was elected leader, sending a full tablet report...
I20260928 04:43:36.715770 9298 tablet_bootstrap.cc:492] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51: No bootstrap required, opened a new log
I20260928 04:43:36.715847 9298 ts_tablet_manager.cc:1397] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 04:43:36.716508 9271 heartbeater.cc:503] Master 127.1.226.254:42709 was elected leader, sending a full tablet report...
I20260928 04:43:36.718463 9298 raft_consensus.cc:348] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554061b51b954cdf83cd7da7df877b51" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43207 } } peers { permanent_uuid: "fc4ea0c4703a4ab998401c9357d616da" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 33589 } } peers { permanent_uuid: "033aee8210cc40bf922b90e5199137c4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45509 } }
I20260928 04:43:36.718586 9298 raft_consensus.cc:374] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:36.718621 9298 raft_consensus.cc:729] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 554061b51b954cdf83cd7da7df877b51, State: Initialized, Role: FOLLOWER
I20260928 04:43:36.718737 9298 consensus_queue.cc:260] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554061b51b954cdf83cd7da7df877b51" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43207 } } peers { permanent_uuid: "fc4ea0c4703a4ab998401c9357d616da" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 33589 } } peers { permanent_uuid: "033aee8210cc40bf922b90e5199137c4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45509 } }
I20260928 04:43:36.719004 9298 ts_tablet_manager.cc:1428] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:43:36.719040 8998 heartbeater.cc:503] Master 127.1.226.254:42709 was elected leader, sending a full tablet report...
W20260928 04:43:36.761622 9137 tablet.cc:2369] T 618be3997d304b0fa4f2a4f77867bb19 P fc4ea0c4703a4ab998401c9357d616da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:43:36.777935 9302 raft_consensus.cc:482] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:43:36.778062 9302 raft_consensus.cc:504] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554061b51b954cdf83cd7da7df877b51" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43207 } } peers { permanent_uuid: "fc4ea0c4703a4ab998401c9357d616da" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 33589 } } peers { permanent_uuid: "033aee8210cc40bf922b90e5199137c4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45509 } }
I20260928 04:43:36.778363 9302 leader_election.cc:288] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc4ea0c4703a4ab998401c9357d616da (127.1.226.194:33589), 033aee8210cc40bf922b90e5199137c4 (127.1.226.195:45509)
I20260928 04:43:36.782761 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.778665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:60995 (local address 127.1.226.195:45509)
0928 04:43:36.778708 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.778711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.778877 (+ 166us) server_negotiation.cc:407] Connection header received
0928 04:43:36.778948 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.778950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.778961 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.779021 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:36.779274 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.780021 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.781763 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.782340 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.782419 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.782429 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.782664 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.782703 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:43:36.782760 9293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.778660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:43029 (local address 127.1.226.194:33589)
0928 04:43:36.778708 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.778711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.778723 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:43:36.778974 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.778975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.778985 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.779036 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:36.779274 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.780038 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.781762 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.782347 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.782419 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.782445 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.782644 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.782687 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 04:43:36.782774 9003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.778587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:33589 (local address 127.1.226.193:43029)
0928 04:43:36.778656 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:36.778670 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:36.778722 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:36.779071 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:36.779073 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:36.779075 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:36.779194 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.779195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.780134 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.780135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.781661 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.781665 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.782441 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.782442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.782544 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.782549 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:43:36.782619 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 04:43:36.782678 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":96}
I20260928 04:43:36.783308 9002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.778597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:45509 (local address 127.1.226.193:60995)
0928 04:43:36.778780 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:36.778810 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:36.778888 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:36.779071 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:36.779073 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:36.779075 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:36.779185 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.779188 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.780133 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.780135 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.781663 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.781665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.782441 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.782442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.782544 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.782549 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:43:36.782619 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 04:43:36.782730 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":225}
I20260928 04:43:36.783510 9088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618be3997d304b0fa4f2a4f77867bb19" candidate_uuid: "554061b51b954cdf83cd7da7df877b51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc4ea0c4703a4ab998401c9357d616da" is_pre_election: true
I20260928 04:43:36.783699 9088 raft_consensus.cc:2454] T 618be3997d304b0fa4f2a4f77867bb19 P fc4ea0c4703a4ab998401c9357d616da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 554061b51b954cdf83cd7da7df877b51 in term 0.
I20260928 04:43:36.783893 9224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618be3997d304b0fa4f2a4f77867bb19" candidate_uuid: "554061b51b954cdf83cd7da7df877b51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "033aee8210cc40bf922b90e5199137c4" is_pre_election: true
I20260928 04:43:36.783972 8886 leader_election.cc:302] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 554061b51b954cdf83cd7da7df877b51, fc4ea0c4703a4ab998401c9357d616da; no voters:
I20260928 04:43:36.784057 9224 raft_consensus.cc:2454] T 618be3997d304b0fa4f2a4f77867bb19 P 033aee8210cc40bf922b90e5199137c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 554061b51b954cdf83cd7da7df877b51 in term 0.
I20260928 04:43:36.784134 9302 raft_consensus.cc:2781] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:43:36.784240 9302 raft_consensus.cc:482] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:43:36.784261 9302 raft_consensus.cc:3037] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:36.785022 9302 raft_consensus.cc:504] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554061b51b954cdf83cd7da7df877b51" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43207 } } peers { permanent_uuid: "fc4ea0c4703a4ab998401c9357d616da" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 33589 } } peers { permanent_uuid: "033aee8210cc40bf922b90e5199137c4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45509 } }
I20260928 04:43:36.785161 9302 leader_election.cc:288] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51 [CANDIDATE]: Term 1 election: Requested vote from peers fc4ea0c4703a4ab998401c9357d616da (127.1.226.194:33589), 033aee8210cc40bf922b90e5199137c4 (127.1.226.195:45509)
I20260928 04:43:36.785311 9088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618be3997d304b0fa4f2a4f77867bb19" candidate_uuid: "554061b51b954cdf83cd7da7df877b51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc4ea0c4703a4ab998401c9357d616da"
I20260928 04:43:36.785382 9224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618be3997d304b0fa4f2a4f77867bb19" candidate_uuid: "554061b51b954cdf83cd7da7df877b51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "033aee8210cc40bf922b90e5199137c4"
I20260928 04:43:36.785419 9088 raft_consensus.cc:3037] T 618be3997d304b0fa4f2a4f77867bb19 P fc4ea0c4703a4ab998401c9357d616da [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:36.785449 9224 raft_consensus.cc:3037] T 618be3997d304b0fa4f2a4f77867bb19 P 033aee8210cc40bf922b90e5199137c4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:36.786346 9224 raft_consensus.cc:2454] T 618be3997d304b0fa4f2a4f77867bb19 P 033aee8210cc40bf922b90e5199137c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 554061b51b954cdf83cd7da7df877b51 in term 1.
I20260928 04:43:36.786346 9088 raft_consensus.cc:2454] T 618be3997d304b0fa4f2a4f77867bb19 P fc4ea0c4703a4ab998401c9357d616da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 554061b51b954cdf83cd7da7df877b51 in term 1.
I20260928 04:43:36.786530 8886 leader_election.cc:302] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 554061b51b954cdf83cd7da7df877b51, fc4ea0c4703a4ab998401c9357d616da; no voters:
I20260928 04:43:36.786634 9302 raft_consensus.cc:2781] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:36.786847 9302 raft_consensus.cc:686] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51 [term 1 LEADER]: Becoming Leader. State: Replica: 554061b51b954cdf83cd7da7df877b51, State: Running, Role: LEADER
I20260928 04:43:36.786962 9302 consensus_queue.cc:237] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554061b51b954cdf83cd7da7df877b51" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43207 } } peers { permanent_uuid: "fc4ea0c4703a4ab998401c9357d616da" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 33589 } } peers { permanent_uuid: "033aee8210cc40bf922b90e5199137c4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45509 } }
I20260928 04:43:36.787976 8032 catalog_manager.cc:5495] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 554061b51b954cdf83cd7da7df877b51 (127.1.226.193). New cstate: current_term: 1 leader_uuid: "554061b51b954cdf83cd7da7df877b51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554061b51b954cdf83cd7da7df877b51" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc4ea0c4703a4ab998401c9357d616da" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 33589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "033aee8210cc40bf922b90e5199137c4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45509 } health_report { overall_health: UNKNOWN } } }
I20260928 04:43:36.797979 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.795081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:45505 (local address 127.0.0.1:37794)
0928 04:43:36.795156 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:36.795172 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:36.795209 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:36.795433 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:36.795433 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:36.795435 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:36.795480 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.795482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.796298 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.796298 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.797088 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:36.797091 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.797756 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.797757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:36.797815 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.797818 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:43:36.797872 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 04:43:36.797919 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 04:43:36.798002 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.795189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37794 (local address 127.1.226.252:45505)
0928 04:43:36.795268 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.795270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.795279 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:43:36.795303 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.795303 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.795324 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.795396 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:36.795578 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.796146 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.797187 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.797646 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.797719 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.797778 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.797904 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.797941 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 04:43:36.798360 8828 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.10] Dedup: 1.7->[1.8-1.10]
I20260928 04:43:36.799222 8860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab16fb1ab1aa488299d5049c1f16fa81. Latest consensus state: current_term: 1 leader_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: NON_VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: true } } }
I20260928 04:43:36.799330 8860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [sys.catalog]: This master's current role is: LEARNER
I20260928 04:43:36.804122 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.801927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51088 (local address 127.1.226.193:43207)
0928 04:43:36.801986 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.801988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.801996 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:43:36.802027 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.802027 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.802044 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.802103 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:43:36.802214 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.802661 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.803436 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.803610 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.803660 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.803822 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:43:36.803913 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:43:36.803977 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.804018 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.804035 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 04:43:36.806242 9088 raft_consensus.cc:1261] T 618be3997d304b0fa4f2a4f77867bb19 P fc4ea0c4703a4ab998401c9357d616da [term 1 FOLLOWER]: Refusing update from remote peer 554061b51b954cdf83cd7da7df877b51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:36.806242 9224 raft_consensus.cc:1261] T 618be3997d304b0fa4f2a4f77867bb19 P 033aee8210cc40bf922b90e5199137c4 [term 1 FOLLOWER]: Refusing update from remote peer 554061b51b954cdf83cd7da7df877b51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:36.806675 9303 consensus_queue.cc:1035] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc4ea0c4703a4ab998401c9357d616da" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 33589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:36.806811 9302 consensus_queue.cc:1035] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "033aee8210cc40bf922b90e5199137c4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:36.809360 9309 mvcc.cc:205] Tried to move back new op lower bound from 7334177246434144256 to 7334177246359846912. Current Snapshot: MvccSnapshot[applied={T|T < 7334177246434144256}]
I20260928 04:43:36.809873 9310 mvcc.cc:205] Tried to move back new op lower bound from 7334177246434144256 to 7334177246359846912. Current Snapshot: MvccSnapshot[applied={T|T < 7334177246434144256}]
I20260928 04:43:36.810775 9311 mvcc.cc:205] Tried to move back new op lower bound from 7334177246434144256 to 7334177246359846912. Current Snapshot: MvccSnapshot[applied={T|T < 7334177246434144256}]
I20260928 04:43:36.813599 8033 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:43:36.813705 8033 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:43:36.814286 8781 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: NON_VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: true } } }
I20260928 04:43:36.814378 8781 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:36.815176 8781 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81: attempting to promote NON_VOTER 2aec508a2bbf4925857d0d1a1d010956 to VOTER
I20260928 04:43:36.815316 8781 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: false } }
I20260928 04:43:36.815744 8828 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [term 1 LEARNER]: Refusing update from remote peer ab16fb1ab1aa488299d5049c1f16fa81: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20260928 04:43:36.815770 8641 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [term 1 FOLLOWER]: Refusing update from remote peer ab16fb1ab1aa488299d5049c1f16fa81: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20260928 04:43:36.815887 8033 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81: Sending DeleteTablet for 3 replicas of tablet 618be3997d304b0fa4f2a4f77867bb19
I20260928 04:43:36.815958 8782 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 04:43:36.816239 9068 tablet_service.cc:1514] Processing DeleteTablet for tablet 618be3997d304b0fa4f2a4f77867bb19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:36 UTC) from {username='oryx', principal='oryx/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:44874
I20260928 04:43:36.816329 8931 tablet_service.cc:1514] Processing DeleteTablet for tablet 618be3997d304b0fa4f2a4f77867bb19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:36 UTC) from {username='oryx', principal='oryx/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:51086
I20260928 04:43:36.816478 9323 tablet_replica.cc:330] T 618be3997d304b0fa4f2a4f77867bb19 P fc4ea0c4703a4ab998401c9357d616da: stopping tablet replica
I20260928 04:43:36.816576 9323 raft_consensus.cc:2229] T 618be3997d304b0fa4f2a4f77867bb19 P fc4ea0c4703a4ab998401c9357d616da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:43:36.816586 9204 tablet_service.cc:1514] Processing DeleteTablet for tablet 618be3997d304b0fa4f2a4f77867bb19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:36 UTC) from {username='oryx', principal='oryx/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:47944
I20260928 04:43:36.816684 9323 raft_consensus.cc:2258] T 618be3997d304b0fa4f2a4f77867bb19 P fc4ea0c4703a4ab998401c9357d616da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:36.816989 9325 tablet_replica.cc:330] T 618be3997d304b0fa4f2a4f77867bb19 P 033aee8210cc40bf922b90e5199137c4: stopping tablet replica
I20260928 04:43:36.817123 9325 raft_consensus.cc:2229] T 618be3997d304b0fa4f2a4f77867bb19 P 033aee8210cc40bf922b90e5199137c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:43:36.817183 9323 ts_tablet_manager.cc:1905] T 618be3997d304b0fa4f2a4f77867bb19 P fc4ea0c4703a4ab998401c9357d616da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:36.817315 9325 raft_consensus.cc:2258] T 618be3997d304b0fa4f2a4f77867bb19 P 033aee8210cc40bf922b90e5199137c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:36.817765 9325 ts_tablet_manager.cc:1905] T 618be3997d304b0fa4f2a4f77867bb19 P 033aee8210cc40bf922b90e5199137c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:36.817912 9320 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260928 04:43:36.818373 8782 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab16fb1ab1aa488299d5049c1f16fa81 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 7 to 12, 2aec508a2bbf4925857d0d1a1d010956 (127.1.226.252) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: false } } }
I20260928 04:43:36.818439 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 8001
I20260928 04:43:36.819432 9323 ts_tablet_manager.cc:1918] T 618be3997d304b0fa4f2a4f77867bb19 P fc4ea0c4703a4ab998401c9357d616da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:43:36.819514 9323 log.cc:1196] T 618be3997d304b0fa4f2a4f77867bb19 P fc4ea0c4703a4ab998401c9357d616da: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/wal/wals/618be3997d304b0fa4f2a4f77867bb19
I20260928 04:43:36.819802 9323 ts_tablet_manager.cc:1939] T 618be3997d304b0fa4f2a4f77867bb19 P fc4ea0c4703a4ab998401c9357d616da: Deleting consensus metadata
I20260928 04:43:36.821127 9325 ts_tablet_manager.cc:1918] T 618be3997d304b0fa4f2a4f77867bb19 P 033aee8210cc40bf922b90e5199137c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:43:36.821218 9325 log.cc:1196] T 618be3997d304b0fa4f2a4f77867bb19 P 033aee8210cc40bf922b90e5199137c4: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/wal/wals/618be3997d304b0fa4f2a4f77867bb19
I20260928 04:43:36.821460 9325 ts_tablet_manager.cc:1939] T 618be3997d304b0fa4f2a4f77867bb19 P 033aee8210cc40bf922b90e5199137c4: Deleting consensus metadata
I20260928 04:43:36.823282 9324 tablet_replica.cc:330] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51: stopping tablet replica
I20260928 04:43:36.823422 9324 raft_consensus.cc:2229] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:43:36.823568 9324 raft_consensus.cc:2258] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:36.824455 9324 ts_tablet_manager.cc:1905] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:36.826996 9324 ts_tablet_manager.cc:1918] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:43:36.827099 9324 log.cc:1196] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-0/wal/wals/618be3997d304b0fa4f2a4f77867bb19
I20260928 04:43:36.827404 9324 ts_tablet_manager.cc:1939] T 618be3997d304b0fa4f2a4f77867bb19 P 554061b51b954cdf83cd7da7df877b51: Deleting consensus metadata
I20260928 04:43:36.828430 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.817273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37808 (local address 127.1.226.252:45505)
0928 04:43:36.817486 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.817488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.819864 (+ 2376us) server_negotiation.cc:407] Connection header received
0928 04:43:36.820015 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.820016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.820028 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.820063 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:36.821384 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.822159 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.823178 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.823425 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.823510 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.825278 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:36.825280 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:36.825280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:36.825288 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:36.827181 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.827563 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.827564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.827565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.827623 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.827884 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.827885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.827885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.828001 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:36.828104 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.828241 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.828322 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
W20260928 04:43:36.833736 9022 connection.cc:535] server connection from 127.0.0.1:44874 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 04:43:36.834206 9160 connection.cc:535] server connection from 127.0.0.1:47944 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 04:43:36.834362 8887 connection.cc:535] server connection from 127.0.0.1:51086 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 04:43:36.843801 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.835741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37812 (local address 127.1.226.252:45505)
0928 04:43:36.835787 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.835789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.835847 (+ 58us) server_negotiation.cc:407] Connection header received
0928 04:43:36.835928 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.835929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.835939 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.835984 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:36.836806 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.837589 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.838738 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.839022 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.839084 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.840648 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:36.840650 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:36.840650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:36.840657 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:36.842451 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.842854 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.842855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.842856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.842919 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.843125 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.843126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.843126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.843293 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:36.843394 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.843613 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.843698 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:43:36.844403 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:36.835423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34038 (local address 127.1.226.253:40789)
0928 04:43:36.835479 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:36.835481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:36.835676 (+ 195us) server_negotiation.cc:407] Connection header received
0928 04:43:36.835788 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:36.835789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:36.835802 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:36.835832 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:36.836932 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.837638 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.838463 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:36.838687 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:36.838773 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:36.840639 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:36.840641 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:36.840641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:36.840650 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:36.842817 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.843251 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.843253 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.843253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.843308 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:36.843532 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:36.843533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:36.843533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:36.843678 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:36.843767 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:36.844217 (+ 450us) server_negotiation.cc:299] Negotiation successful
0928 04:43:36.844296 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 04:43:37.811595 9333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:37.811289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.1.226.194:49175)
0928 04:43:37.811441 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:37.811534 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":73,"threads_started":1}
W20260928 04:43:37.811982 9135 heartbeater.cc:644] Failed to heartbeat to 127.1.226.254:42709 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
I20260928 04:43:37.812243 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:37.811966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.1.226.195:47363)
0928 04:43:37.812083 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:37.812196 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":55,"threads_started":1}
I20260928 04:43:37.812301 9334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:37.811872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.1.226.193:58893)
0928 04:43:37.812186 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:37.812267 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":52,"threads_started":1}
W20260928 04:43:37.812482 8998 heartbeater.cc:644] Failed to heartbeat to 127.1.226.254:42709 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
W20260928 04:43:37.812484 9271 heartbeater.cc:644] Failed to heartbeat to 127.1.226.254:42709 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
I20260928 04:43:38.316123 9336 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab16fb1ab1aa488299d5049c1f16fa81)
I20260928 04:43:38.316361 9336 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: NON_VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: true } }
I20260928 04:43:38.316640 9336 leader_election.cc:288] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab16fb1ab1aa488299d5049c1f16fa81 (127.1.226.254:42709)
I20260928 04:43:38.317065 9337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.316820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.0.0.1:41002)
0928 04:43:38.316969 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:38.317028 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
W20260928 04:43:38.317195 8617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
I20260928 04:43:38.317580 9337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.317478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.0.0.1:41004)
0928 04:43:38.317538 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:38.317564 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 04:43:38.317679 8617 leader_election.cc:334] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ab16fb1ab1aa488299d5049c1f16fa81 (127.1.226.254:42709): Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
I20260928 04:43:38.317741 8617 leader_election.cc:302] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 8030293b08fe463b9af2092f0c6a7233; no voters: ab16fb1ab1aa488299d5049c1f16fa81
I20260928 04:43:38.317858 9336 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 04:43:38.328047 9339 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab16fb1ab1aa488299d5049c1f16fa81)
I20260928 04:43:38.328293 9339 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: false } }
I20260928 04:43:38.328671 9339 leader_election.cc:288] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab16fb1ab1aa488299d5049c1f16fa81 (127.1.226.254:42709), 8030293b08fe463b9af2092f0c6a7233 (127.1.226.253:40789)
I20260928 04:43:38.329059 9341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.328799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.0.0.1:41020)
0928 04:43:38.328946 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:38.329015 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":59,"threads_started":1}
W20260928 04:43:38.329185 8804 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
I20260928 04:43:38.329603 9341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.329495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.0.0.1:41032)
0928 04:43:38.329546 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:38.329575 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 04:43:38.329770 8804 leader_election.cc:334] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ab16fb1ab1aa488299d5049c1f16fa81 (127.1.226.254:42709): Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
I20260928 04:43:38.331985 9340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.328831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:40789 (local address 127.0.0.1:34044)
0928 04:43:38.329017 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:38.329028 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:38.329094 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:38.329430 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:38.329431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:38.329433 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:38.329485 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:38.329487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:38.330440 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:38.330441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:38.331212 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:38.331215 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:38.331803 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:38.331805 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:38.331853 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:38.331857 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:43:38.331903 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 04:43:38.331939 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":68,"threads_started":1}
I20260928 04:43:38.332003 9342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.328919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34044 (local address 127.1.226.253:40789)
0928 04:43:38.329025 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:38.329028 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:38.329104 (+ 76us) server_negotiation.cc:407] Connection header received
0928 04:43:38.329295 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:38.329296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:38.329309 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:38.329377 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:38.329564 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:38.330335 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:38.331303 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:38.331692 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:38.331751 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:38.331760 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:38.331921 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 04:43:38.331951 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":47,"threads_started":1}
I20260928 04:43:38.332276 8641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2aec508a2bbf4925857d0d1a1d010956" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8030293b08fe463b9af2092f0c6a7233" is_pre_election: true
I20260928 04:43:38.332398 8641 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2aec508a2bbf4925857d0d1a1d010956 in term 1.
I20260928 04:43:38.332624 8805 leader_election.cc:302] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aec508a2bbf4925857d0d1a1d010956, 8030293b08fe463b9af2092f0c6a7233; no voters: ab16fb1ab1aa488299d5049c1f16fa81
I20260928 04:43:38.332767 9339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 04:43:38.332815 9339 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ab16fb1ab1aa488299d5049c1f16fa81)
I20260928 04:43:38.332846 9339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [term 1 FOLLOWER]: Advancing to term 2
I20260928 04:43:38.333530 9339 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: false } }
I20260928 04:43:38.333680 9339 leader_election.cc:288] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [CANDIDATE]: Term 2 election: Requested vote from peers ab16fb1ab1aa488299d5049c1f16fa81 (127.1.226.254:42709), 8030293b08fe463b9af2092f0c6a7233 (127.1.226.253:40789)
I20260928 04:43:38.333835 9340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.333746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.0.0.1:41038)
0928 04:43:38.333790 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:38.333813 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 04:43:38.333873 8641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2aec508a2bbf4925857d0d1a1d010956" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8030293b08fe463b9af2092f0c6a7233"
I20260928 04:43:38.333949 8641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [term 1 FOLLOWER]: Advancing to term 2
I20260928 04:43:38.334193 9340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.334039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.0.0.1:41044)
0928 04:43:38.334137 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:38.334166 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260928 04:43:38.334293 8804 leader_election.cc:334] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ab16fb1ab1aa488299d5049c1f16fa81 (127.1.226.254:42709): Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
I20260928 04:43:38.334707 8641 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aec508a2bbf4925857d0d1a1d010956 in term 2.
I20260928 04:43:38.334859 8805 leader_election.cc:302] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2aec508a2bbf4925857d0d1a1d010956, 8030293b08fe463b9af2092f0c6a7233; no voters: ab16fb1ab1aa488299d5049c1f16fa81
I20260928 04:43:38.334970 9339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 04:43:38.335129 9339 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [term 2 LEADER]: Becoming Leader. State: Replica: 2aec508a2bbf4925857d0d1a1d010956, State: Running, Role: LEADER
I20260928 04:43:38.335218 9339 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: false } }
I20260928 04:43:38.335587 9344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2aec508a2bbf4925857d0d1a1d010956. Latest consensus state: current_term: 2 leader_uuid: "2aec508a2bbf4925857d0d1a1d010956" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: NON_VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: true } } } pending_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: false } } }
I20260928 04:43:38.335673 9344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:38.800657 9340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.800475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.0.0.1:41128)
0928 04:43:38.800560 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:38.800609 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 04:43:38.801070 9340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.800976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.0.0.1:41136)
0928 04:43:38.801020 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:38.801048 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 04:43:38.801213 8804 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 -> Peer ab16fb1ab1aa488299d5049c1f16fa81 (127.1.226.254:42709): Couldn't send request to peer ab16fb1ab1aa488299d5049c1f16fa81. Status: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 04:43:38.803220 8641 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [term 2 FOLLOWER]: Refusing update from remote peer 2aec508a2bbf4925857d0d1a1d010956: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20260928 04:43:38.803543 9344 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260928 04:43:38.804462 9336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2aec508a2bbf4925857d0d1a1d010956. Latest consensus state: current_term: 2 leader_uuid: "2aec508a2bbf4925857d0d1a1d010956" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: NON_VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: true } } } pending_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: false } } }
I20260928 04:43:38.804600 9336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:43:38.804970 9339 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [term 2 LEADER]: Committing config change with OpId 1.12: config changed from index 7 to 12, 2aec508a2bbf4925857d0d1a1d010956 (127.1.226.252) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: false } } }
I20260928 04:43:38.805217 8641 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [term 2 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 7 to 12, 2aec508a2bbf4925857d0d1a1d010956 (127.1.226.252) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: false } } }
I20260928 04:43:38.806211 9344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "2aec508a2bbf4925857d0d1a1d010956" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: false } } }
I20260928 04:43:38.806255 9336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "2aec508a2bbf4925857d0d1a1d010956" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: false } } }
I20260928 04:43:38.806331 9336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:43:38.806345 9344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:38.806224 9339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "2aec508a2bbf4925857d0d1a1d010956" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: false } } }
I20260928 04:43:38.806383 9339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:38.806380 9336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "2aec508a2bbf4925857d0d1a1d010956" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ab16fb1ab1aa488299d5049c1f16fa81" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42709 } } peers { permanent_uuid: "8030293b08fe463b9af2092f0c6a7233" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 40789 } attrs { promote: false } } peers { permanent_uuid: "2aec508a2bbf4925857d0d1a1d010956" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45505 } attrs { promote: false } } }
I20260928 04:43:38.806439 9336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8030293b08fe463b9af2092f0c6a7233 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:43:38.812665 9353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.812401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.1.226.194:34727)
0928 04:43:38.812570 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:38.812611 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":91,"threads_started":1}
I20260928 04:43:38.813051 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.812880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.1.226.195:60153)
0928 04:43:38.812981 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:38.813016 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":67,"threads_started":1}
I20260928 04:43:38.813292 9355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.813058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.1.226.193:50727)
0928 04:43:38.813208 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:38.813242 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
I20260928 04:43:38.851559 9346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:43:38.852213 9346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:43:38.852308 9346 catalog_manager.cc:1259] Loaded cluster ID: 2e45c915021c4b2b9faa0da3abf6c355
I20260928 04:43:38.852362 9346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:43:38.852584 9346 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:43:38.852686 9346 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956: Loaded TSK: 0
I20260928 04:43:38.852846 9346 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:43:38.919368 8817 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:43:38.921060 8817 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:43:38.928299 9361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.924994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51102 (local address 127.1.226.193:43207)
0928 04:43:38.925333 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:38.925335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:38.925344 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:43:38.925370 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:38.925370 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:38.925382 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:38.925456 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:38.925572 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:38.926049 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:38.927451 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:38.927873 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:38.927974 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:38.927986 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:38.928147 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 04:43:38.928229 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":57,"threads_started":1}
I20260928 04:43:38.928298 9340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.924713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:43207 (local address 127.0.0.1:51102)
0928 04:43:38.924819 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:38.924833 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:38.924886 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:38.925458 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:38.925459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:38.925460 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:38.925504 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:38.925506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:38.926162 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:38.926163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:38.927333 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:38.927336 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:38.927974 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:38.927975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:38.928037 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:38.928042 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:43:38.928139 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 04:43:38.928216 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 04:43:38.928314 9360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.924896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47952 (local address 127.1.226.195:45509)
0928 04:43:38.925068 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:38.925071 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:38.925110 (+ 39us) server_negotiation.cc:407] Connection header received
0928 04:43:38.925142 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:38.925143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:38.925156 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:38.925252 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:38.925457 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:38.926224 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:38.927337 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:38.927901 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:38.927967 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:38.927979 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:38.928182 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 04:43:38.928226 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":59,"threads_started":1}
I20260928 04:43:38.928429 9357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.924802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:33589 (local address 127.0.0.1:44884)
0928 04:43:38.925100 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:38.925110 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:38.925172 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:38.925372 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:38.925372 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:38.925374 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:38.925407 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:38.925409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:38.926309 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:38.926310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:38.927428 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:38.927431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:38.928209 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:38.928210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:38.928267 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:38.928271 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:43:38.928315 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 04:43:38.928359 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":22,"thread_start_us":70,"threads_started":1}
I20260928 04:43:38.928525 9358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.924729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:45509 (local address 127.0.0.1:47952)
0928 04:43:38.924988 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:38.924998 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:38.925046 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:38.925276 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:38.925277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:38.925279 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:38.925359 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:38.925361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:38.926315 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:38.926316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:38.927248 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:38.927250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:38.928011 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:38.928012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:38.928074 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:38.928078 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:43:38.928394 (+ 316us) client_negotiation.cc:240] Negotiation successful
0928 04:43:38.928429 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":4,"thread_start_us":55,"threads_started":1}
I20260928 04:43:38.928668 9359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.924877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44884 (local address 127.1.226.194:33589)
0928 04:43:38.924984 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:38.924986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:38.925159 (+ 173us) server_negotiation.cc:407] Connection header received
0928 04:43:38.925264 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:38.925266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:38.925278 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:38.925362 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:38.925491 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:38.926217 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:38.927513 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:38.928075 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:38.928170 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:38.928181 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:38.928523 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 04:43:38.928573 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":60,"threads_started":1}
I20260928 04:43:38.928797 9204 tablet_service.cc:1467] Processing CreateTablet for tablet 12bfeee5b59e4533a16549db9efc4d21 (DEFAULT_TABLE table=test-table [id=f10ea26e5e8543efa2394032d0706eec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:38.928875 9068 tablet_service.cc:1467] Processing CreateTablet for tablet 12bfeee5b59e4533a16549db9efc4d21 (DEFAULT_TABLE table=test-table [id=f10ea26e5e8543efa2394032d0706eec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:38.928944 9204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12bfeee5b59e4533a16549db9efc4d21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:38.929003 9068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12bfeee5b59e4533a16549db9efc4d21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:38.929445 8931 tablet_service.cc:1467] Processing CreateTablet for tablet 12bfeee5b59e4533a16549db9efc4d21 (DEFAULT_TABLE table=test-table [id=f10ea26e5e8543efa2394032d0706eec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:38.929564 8931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12bfeee5b59e4533a16549db9efc4d21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:38.930361 9362 tablet_bootstrap.cc:492] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4: Bootstrap starting.
I20260928 04:43:38.930480 9363 tablet_bootstrap.cc:492] T 12bfeee5b59e4533a16549db9efc4d21 P fc4ea0c4703a4ab998401c9357d616da: Bootstrap starting.
I20260928 04:43:38.930886 9363 tablet_bootstrap.cc:654] T 12bfeee5b59e4533a16549db9efc4d21 P fc4ea0c4703a4ab998401c9357d616da: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:38.930886 9362 tablet_bootstrap.cc:654] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:38.931201 9364 tablet_bootstrap.cc:492] T 12bfeee5b59e4533a16549db9efc4d21 P 554061b51b954cdf83cd7da7df877b51: Bootstrap starting.
I20260928 04:43:38.931625 9364 tablet_bootstrap.cc:654] T 12bfeee5b59e4533a16549db9efc4d21 P 554061b51b954cdf83cd7da7df877b51: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:38.931710 9362 tablet_bootstrap.cc:492] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4: No bootstrap required, opened a new log
I20260928 04:43:38.931766 9362 ts_tablet_manager.cc:1397] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:38.931903 9362 raft_consensus.cc:348] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554061b51b954cdf83cd7da7df877b51" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43207 } } peers { permanent_uuid: "033aee8210cc40bf922b90e5199137c4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45509 } } peers { permanent_uuid: "fc4ea0c4703a4ab998401c9357d616da" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 33589 } }
I20260928 04:43:38.931998 9362 raft_consensus.cc:374] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:38.932029 9362 raft_consensus.cc:729] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 033aee8210cc40bf922b90e5199137c4, State: Initialized, Role: FOLLOWER
I20260928 04:43:38.932112 9362 consensus_queue.cc:260] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554061b51b954cdf83cd7da7df877b51" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43207 } } peers { permanent_uuid: "033aee8210cc40bf922b90e5199137c4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45509 } } peers { permanent_uuid: "fc4ea0c4703a4ab998401c9357d616da" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 33589 } }
I20260928 04:43:38.932233 9364 tablet_bootstrap.cc:492] T 12bfeee5b59e4533a16549db9efc4d21 P 554061b51b954cdf83cd7da7df877b51: No bootstrap required, opened a new log
I20260928 04:43:38.932282 9364 ts_tablet_manager.cc:1397] T 12bfeee5b59e4533a16549db9efc4d21 P 554061b51b954cdf83cd7da7df877b51: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:38.932305 9363 tablet_bootstrap.cc:492] T 12bfeee5b59e4533a16549db9efc4d21 P fc4ea0c4703a4ab998401c9357d616da: No bootstrap required, opened a new log
I20260928 04:43:38.932353 9363 ts_tablet_manager.cc:1397] T 12bfeee5b59e4533a16549db9efc4d21 P fc4ea0c4703a4ab998401c9357d616da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:43:38.932361 9362 ts_tablet_manager.cc:1428] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:38.932415 9364 raft_consensus.cc:348] T 12bfeee5b59e4533a16549db9efc4d21 P 554061b51b954cdf83cd7da7df877b51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554061b51b954cdf83cd7da7df877b51" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43207 } } peers { permanent_uuid: "033aee8210cc40bf922b90e5199137c4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45509 } } peers { permanent_uuid: "fc4ea0c4703a4ab998401c9357d616da" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 33589 } }
I20260928 04:43:38.932489 9364 raft_consensus.cc:374] T 12bfeee5b59e4533a16549db9efc4d21 P 554061b51b954cdf83cd7da7df877b51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:38.932513 9364 raft_consensus.cc:729] T 12bfeee5b59e4533a16549db9efc4d21 P 554061b51b954cdf83cd7da7df877b51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 554061b51b954cdf83cd7da7df877b51, State: Initialized, Role: FOLLOWER
I20260928 04:43:38.932543 9363 raft_consensus.cc:348] T 12bfeee5b59e4533a16549db9efc4d21 P fc4ea0c4703a4ab998401c9357d616da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554061b51b954cdf83cd7da7df877b51" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43207 } } peers { permanent_uuid: "033aee8210cc40bf922b90e5199137c4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45509 } } peers { permanent_uuid: "fc4ea0c4703a4ab998401c9357d616da" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 33589 } }
I20260928 04:43:38.932578 9364 consensus_queue.cc:260] T 12bfeee5b59e4533a16549db9efc4d21 P 554061b51b954cdf83cd7da7df877b51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554061b51b954cdf83cd7da7df877b51" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43207 } } peers { permanent_uuid: "033aee8210cc40bf922b90e5199137c4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45509 } } peers { permanent_uuid: "fc4ea0c4703a4ab998401c9357d616da" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 33589 } }
I20260928 04:43:38.932632 9363 raft_consensus.cc:374] T 12bfeee5b59e4533a16549db9efc4d21 P fc4ea0c4703a4ab998401c9357d616da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:38.932667 9363 raft_consensus.cc:729] T 12bfeee5b59e4533a16549db9efc4d21 P fc4ea0c4703a4ab998401c9357d616da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc4ea0c4703a4ab998401c9357d616da, State: Initialized, Role: FOLLOWER
I20260928 04:43:38.932717 9363 consensus_queue.cc:260] T 12bfeee5b59e4533a16549db9efc4d21 P fc4ea0c4703a4ab998401c9357d616da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554061b51b954cdf83cd7da7df877b51" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43207 } } peers { permanent_uuid: "033aee8210cc40bf922b90e5199137c4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45509 } } peers { permanent_uuid: "fc4ea0c4703a4ab998401c9357d616da" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 33589 } }
I20260928 04:43:38.932749 9364 ts_tablet_manager.cc:1428] T 12bfeee5b59e4533a16549db9efc4d21 P 554061b51b954cdf83cd7da7df877b51: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 04:43:38.932835 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.932724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.1.226.195:39675)
0928 04:43:38.932770 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:38.932808 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 04:43:38.932924 9363 ts_tablet_manager.cc:1428] T 12bfeee5b59e4533a16549db9efc4d21 P fc4ea0c4703a4ab998401c9357d616da: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 04:43:38.933547 9355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.933418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.1.226.193:59139)
0928 04:43:38.933483 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:38.933518 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 04:43:38.933960 9353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.933700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.1.226.194:45083)
0928 04:43:38.933887 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:38.933922 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
I20260928 04:43:38.934166 9134 heartbeater.cc:503] Master 127.1.226.252:45505 was elected leader, sending a full tablet report...
I20260928 04:43:38.934194 9270 heartbeater.cc:503] Master 127.1.226.252:45505 was elected leader, sending a full tablet report...
I20260928 04:43:38.934404 9353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.934315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.1.226.194:57443)
0928 04:43:38.934356 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:38.934381 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 04:43:38.934921 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.934802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.1.226.195:34143)
0928 04:43:38.934859 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:38.934889 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 04:43:38.934969 8997 heartbeater.cc:503] Master 127.1.226.252:45505 was elected leader, sending a full tablet report...
I20260928 04:43:38.935240 9355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.935083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.1.226.193:41345)
0928 04:43:38.935185 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:38.935212 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 04:43:38.987254 9368 raft_consensus.cc:482] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:43:38.987396 9368 raft_consensus.cc:504] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554061b51b954cdf83cd7da7df877b51" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43207 } } peers { permanent_uuid: "033aee8210cc40bf922b90e5199137c4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45509 } } peers { permanent_uuid: "fc4ea0c4703a4ab998401c9357d616da" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 33589 } }
I20260928 04:43:38.987805 9368 leader_election.cc:288] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 554061b51b954cdf83cd7da7df877b51 (127.1.226.193:43207), fc4ea0c4703a4ab998401c9357d616da (127.1.226.194:33589)
I20260928 04:43:38.991146 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.988042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:43207 (local address 127.1.226.195:39759)
0928 04:43:38.988117 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:38.988131 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:38.988240 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:38.988381 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:38.988382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:38.988384 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:38.988462 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:38.988464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:38.989213 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:38.989214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:38.990165 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:38.990167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:38.990849 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:38.990850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:38.990923 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:38.990928 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:43:38.990990 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 04:43:38.991031 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 04:43:38.991190 9361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.988128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:39759 (local address 127.1.226.193:43207)
0928 04:43:38.988202 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:38.988204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:38.988221 (+ 17us) server_negotiation.cc:407] Connection header received
0928 04:43:38.988303 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:38.988304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:38.988315 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:38.988367 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:38.988540 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:38.989135 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:38.990262 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:38.990775 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:38.990830 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:38.990838 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:38.991039 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 04:43:38.991082 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 04:43:38.991415 8951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12bfeee5b59e4533a16549db9efc4d21" candidate_uuid: "033aee8210cc40bf922b90e5199137c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "554061b51b954cdf83cd7da7df877b51" is_pre_election: true
I20260928 04:43:38.991508 9359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.988287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:57267 (local address 127.1.226.194:33589)
0928 04:43:38.988343 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:38.988345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:38.988491 (+ 146us) server_negotiation.cc:407] Connection header received
0928 04:43:38.988582 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:38.988583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:38.988617 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:38.988733 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:38.988917 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:38.989543 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:38.990576 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:38.991063 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:38.991127 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:38.991138 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:38.991407 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 04:43:38.991444 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 04:43:38.991552 8951 raft_consensus.cc:2454] T 12bfeee5b59e4533a16549db9efc4d21 P 554061b51b954cdf83cd7da7df877b51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 033aee8210cc40bf922b90e5199137c4 in term 0.
I20260928 04:43:38.991581 9371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.988225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:33589 (local address 127.1.226.195:57267)
0928 04:43:38.988388 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:38.988402 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:38.988484 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:38.988776 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:38.988777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:38.988779 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:38.988840 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:38.988842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:38.989631 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:38.989632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:38.990480 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:38.990482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:38.991185 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:38.991186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:38.991264 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:38.991268 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:43:38.991404 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 04:43:38.991440 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":117,"threads_started":1}
I20260928 04:43:38.991798 9159 leader_election.cc:302] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 033aee8210cc40bf922b90e5199137c4, 554061b51b954cdf83cd7da7df877b51; no voters:
I20260928 04:43:38.991853 9088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12bfeee5b59e4533a16549db9efc4d21" candidate_uuid: "033aee8210cc40bf922b90e5199137c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc4ea0c4703a4ab998401c9357d616da" is_pre_election: true
I20260928 04:43:38.991950 9088 raft_consensus.cc:2454] T 12bfeee5b59e4533a16549db9efc4d21 P fc4ea0c4703a4ab998401c9357d616da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 033aee8210cc40bf922b90e5199137c4 in term 0.
I20260928 04:43:38.991948 9368 raft_consensus.cc:2781] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:43:38.991993 9368 raft_consensus.cc:482] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:43:38.992023 9368 raft_consensus.cc:3037] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:38.992913 9368 raft_consensus.cc:504] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554061b51b954cdf83cd7da7df877b51" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43207 } } peers { permanent_uuid: "033aee8210cc40bf922b90e5199137c4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45509 } } peers { permanent_uuid: "fc4ea0c4703a4ab998401c9357d616da" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 33589 } }
I20260928 04:43:38.993034 9368 leader_election.cc:288] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4 [CANDIDATE]: Term 1 election: Requested vote from peers 554061b51b954cdf83cd7da7df877b51 (127.1.226.193:43207), fc4ea0c4703a4ab998401c9357d616da (127.1.226.194:33589)
I20260928 04:43:38.993260 8951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12bfeee5b59e4533a16549db9efc4d21" candidate_uuid: "033aee8210cc40bf922b90e5199137c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "554061b51b954cdf83cd7da7df877b51"
I20260928 04:43:38.993335 8951 raft_consensus.cc:3037] T 12bfeee5b59e4533a16549db9efc4d21 P 554061b51b954cdf83cd7da7df877b51 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:38.993317 9088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12bfeee5b59e4533a16549db9efc4d21" candidate_uuid: "033aee8210cc40bf922b90e5199137c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc4ea0c4703a4ab998401c9357d616da"
I20260928 04:43:38.993402 9088 raft_consensus.cc:3037] T 12bfeee5b59e4533a16549db9efc4d21 P fc4ea0c4703a4ab998401c9357d616da [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:38.994022 8951 raft_consensus.cc:2454] T 12bfeee5b59e4533a16549db9efc4d21 P 554061b51b954cdf83cd7da7df877b51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 033aee8210cc40bf922b90e5199137c4 in term 1.
I20260928 04:43:38.994120 9088 raft_consensus.cc:2454] T 12bfeee5b59e4533a16549db9efc4d21 P fc4ea0c4703a4ab998401c9357d616da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 033aee8210cc40bf922b90e5199137c4 in term 1.
I20260928 04:43:38.994210 9159 leader_election.cc:302] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 033aee8210cc40bf922b90e5199137c4, 554061b51b954cdf83cd7da7df877b51; no voters:
I20260928 04:43:38.994324 9368 raft_consensus.cc:2781] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:38.994493 9368 raft_consensus.cc:686] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4 [term 1 LEADER]: Becoming Leader. State: Replica: 033aee8210cc40bf922b90e5199137c4, State: Running, Role: LEADER
I20260928 04:43:38.994679 9368 consensus_queue.cc:237] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554061b51b954cdf83cd7da7df877b51" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43207 } } peers { permanent_uuid: "033aee8210cc40bf922b90e5199137c4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45509 } } peers { permanent_uuid: "fc4ea0c4703a4ab998401c9357d616da" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 33589 } }
I20260928 04:43:38.994903 9371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:38.994799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.1.226.195:40241)
0928 04:43:38.994846 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:38.994876 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 04:43:38.995563 8818 catalog_manager.cc:5495] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 033aee8210cc40bf922b90e5199137c4 (127.1.226.195). New cstate: current_term: 1 leader_uuid: "033aee8210cc40bf922b90e5199137c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "554061b51b954cdf83cd7da7df877b51" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "033aee8210cc40bf922b90e5199137c4" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 45509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc4ea0c4703a4ab998401c9357d616da" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 33589 } health_report { overall_health: UNKNOWN } } }
I20260928 04:43:39.005069 9360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:39.002429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47968 (local address 127.1.226.195:45509)
0928 04:43:39.002475 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:39.002482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:39.002507 (+ 25us) server_negotiation.cc:407] Connection header received
0928 04:43:39.002589 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:39.002590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:39.002608 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:39.002663 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:43:39.002828 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:39.003299 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:39.004080 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:39.004380 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:39.004453 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:39.004700 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:43:39.004817 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:43:39.004888 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:39.004979 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 04:43:39.005002 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:43:39.006680 9088 raft_consensus.cc:1261] T 12bfeee5b59e4533a16549db9efc4d21 P fc4ea0c4703a4ab998401c9357d616da [term 1 FOLLOWER]: Refusing update from remote peer 033aee8210cc40bf922b90e5199137c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:39.006804 8951 raft_consensus.cc:1261] T 12bfeee5b59e4533a16549db9efc4d21 P 554061b51b954cdf83cd7da7df877b51 [term 1 FOLLOWER]: Refusing update from remote peer 033aee8210cc40bf922b90e5199137c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:39.007019 9368 consensus_queue.cc:1035] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc4ea0c4703a4ab998401c9357d616da" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 33589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:39.007170 9372 consensus_queue.cc:1035] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "554061b51b954cdf83cd7da7df877b51" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:39.008055 9353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:39.007937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.1.226.194:53733)
0928 04:43:39.007982 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:39.008017 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 04:43:39.008793 9355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:39.008632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.1.226.193:60843)
0928 04:43:39.008676 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:39.008707 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 04:43:39.009940 9353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:39.009839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.1.226.194:59301)
0928 04:43:39.009878 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:39.009905 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 04:43:39.010437 9371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:39.010309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.1.226.195:51931)
0928 04:43:39.010356 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:39.010385 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 04:43:39.011574 9355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:39.011451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.1.226.193:48119)
0928 04:43:39.011511 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:39.011543 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 04:43:39.011914 9371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:39.011786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42709 (local address 127.1.226.195:34663)
0928 04:43:39.011843 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:39.011878 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:42709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 04:43:39.013090 8817 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:43:39.013195 8817 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:43:39.014590 8817 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2aec508a2bbf4925857d0d1a1d010956: Sending DeleteTablet for 3 replicas of tablet 12bfeee5b59e4533a16549db9efc4d21
I20260928 04:43:39.014881 8931 tablet_service.cc:1514] Processing DeleteTablet for tablet 12bfeee5b59e4533a16549db9efc4d21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:39 UTC) from {username='oryx', principal='oryx/127.1.226.252@KRBTEST.COM'} at 127.0.0.1:51102
I20260928 04:43:39.014987 9068 tablet_service.cc:1514] Processing DeleteTablet for tablet 12bfeee5b59e4533a16549db9efc4d21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:39 UTC) from {username='oryx', principal='oryx/127.1.226.252@KRBTEST.COM'} at 127.0.0.1:44884
I20260928 04:43:39.015131 9385 tablet_replica.cc:330] T 12bfeee5b59e4533a16549db9efc4d21 P 554061b51b954cdf83cd7da7df877b51: stopping tablet replica
I20260928 04:43:39.015166 9386 tablet_replica.cc:330] T 12bfeee5b59e4533a16549db9efc4d21 P fc4ea0c4703a4ab998401c9357d616da: stopping tablet replica
I20260928 04:43:39.015210 9386 raft_consensus.cc:2229] T 12bfeee5b59e4533a16549db9efc4d21 P fc4ea0c4703a4ab998401c9357d616da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:43:39.015200 9385 raft_consensus.cc:2229] T 12bfeee5b59e4533a16549db9efc4d21 P 554061b51b954cdf83cd7da7df877b51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:43:39.015254 9386 raft_consensus.cc:2258] T 12bfeee5b59e4533a16549db9efc4d21 P fc4ea0c4703a4ab998401c9357d616da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:39.015257 9385 raft_consensus.cc:2258] T 12bfeee5b59e4533a16549db9efc4d21 P 554061b51b954cdf83cd7da7df877b51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:39.015569 9385 ts_tablet_manager.cc:1905] T 12bfeee5b59e4533a16549db9efc4d21 P 554061b51b954cdf83cd7da7df877b51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:39.015586 9204 tablet_service.cc:1514] Processing DeleteTablet for tablet 12bfeee5b59e4533a16549db9efc4d21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:39 UTC) from {username='oryx', principal='oryx/127.1.226.252@KRBTEST.COM'} at 127.0.0.1:47952
I20260928 04:43:39.015813 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 8778
I20260928 04:43:39.016842 9386 ts_tablet_manager.cc:1905] T 12bfeee5b59e4533a16549db9efc4d21 P fc4ea0c4703a4ab998401c9357d616da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:39.016855 9387 tablet_replica.cc:330] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4: stopping tablet replica
I20260928 04:43:39.017002 9387 raft_consensus.cc:2229] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:43:39.017143 9387 raft_consensus.cc:2258] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:39.018636 9386 ts_tablet_manager.cc:1918] T 12bfeee5b59e4533a16549db9efc4d21 P fc4ea0c4703a4ab998401c9357d616da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:43:39.018702 9386 log.cc:1196] T 12bfeee5b59e4533a16549db9efc4d21 P fc4ea0c4703a4ab998401c9357d616da: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-1/wal/wals/12bfeee5b59e4533a16549db9efc4d21
I20260928 04:43:39.018945 9386 ts_tablet_manager.cc:1939] T 12bfeee5b59e4533a16549db9efc4d21 P fc4ea0c4703a4ab998401c9357d616da: Deleting consensus metadata
I20260928 04:43:39.019274 8804 catalog_manager.cc:4841] TS fc4ea0c4703a4ab998401c9357d616da (127.1.226.194:33589): tablet 12bfeee5b59e4533a16549db9efc4d21 (table test-table [id=f10ea26e5e8543efa2394032d0706eec]) successfully deleted
I20260928 04:43:39.021337 9387 ts_tablet_manager.cc:1905] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:39.023304 9387 ts_tablet_manager.cc:1918] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:43:39.023375 9387 log.cc:1196] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570585693787-1931-0/minicluster-data/ts-2/wal/wals/12bfeee5b59e4533a16549db9efc4d21
I20260928 04:43:39.023613 9387 ts_tablet_manager.cc:1939] T 12bfeee5b59e4533a16549db9efc4d21 P 033aee8210cc40bf922b90e5199137c4: Deleting consensus metadata
I20260928 04:43:39.024109 8803 catalog_manager.cc:4841] TS 033aee8210cc40bf922b90e5199137c4 (127.1.226.195:45509): tablet 12bfeee5b59e4533a16549db9efc4d21 (table test-table [id=f10ea26e5e8543efa2394032d0706eec]) successfully deleted
W20260928 04:43:39.024134 9358 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:39.023992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:43207 (local address 127.0.0.1:51112)
0928 04:43:39.024040 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:39.024072 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.193:43207: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 04:43:39.024333 8804 catalog_manager.cc:4584] TS 554061b51b954cdf83cd7da7df877b51 (127.1.226.193:43207): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 12bfeee5b59e4533a16549db9efc4d21: Network error: Client connection negotiation failed: client connection to 127.1.226.193:43207: connect: Connection reset by peer (error 104)
I20260928 04:43:39.024459 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 9005
I20260928 04:43:39.032079 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 9141
I20260928 04:43:39.037904 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 8497
I20260928 04:43:39.042693 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 8685
2026-09-28T04:43:39Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5154 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:43:39 dist-test-slave-j5c3 krb5kdc[9390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:43:39 dist-test-slave-j5c3 krb5kdc[9390](info): set up 2 sockets
Sep 28 04:43:39 dist-test-slave-j5c3 krb5kdc[9390](info): commencing operation
krb5kdc: starting...
W20260928 04:43:40.100696 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:43:40 dist-test-slave-j5c3 krb5kdc[9390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:43:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:43:40Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.254.keytab.
Entry for principal oryx/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.254.keytab.
I20260928 04:43:40.226922 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:45703
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.254.keytab
--principal=oryx/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:45703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:40.308794 9403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:40.309043 9403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:40.309080 9403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:40.310263 9403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:43:40.310333 9403 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 04:43:40.310354 9403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:40.310376 9403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:40.310429 9403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:43:40.310459 9403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:43:40.311985 9403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:45703
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.254.keytab
--principal=oryx/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:45703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:40.312284 9403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:40.312517 9403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:40.317739 9408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:40.317773 9403 server_base.cc:1034] running on GCE node
W20260928 04:43:40.317770 9409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:40.317736 9411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:40.318292 9403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:40.318621 9403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:40.319794 9403 hybrid_clock.cc:648] HybridClock initialized: now 1790570620319771 us; error 40 us; skew 500 ppm
Sep 28 04:43:40 dist-test-slave-j5c3 krb5kdc[9390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570620, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:40.322428 9403 init.cc:377] Logged in from keytab as oryx/127.1.226.254@KRBTEST.COM (short username oryx)
I20260928 04:43:40.322816 9403 webserver.cc:460] Webserver started at http://127.1.226.254:34479/ using document root <none> and password file <none>
I20260928 04:43:40.323014 9403 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:40.323053 9403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:40.323138 9403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:40.324138 9403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "7c4ba4735141419a8026489a0785daa9"
format_stamp: "Formatted at 2026-09-28 04:43:40 on dist-test-slave-j5c3"
server_key: "f986926288ec79a012b012a8e6c98a80"
server_key_iv: "5bbc485cf19ae895eac5f02befe10838"
server_key_version: "encryptionkey@0"
I20260928 04:43:40.324520 9403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "7c4ba4735141419a8026489a0785daa9"
format_stamp: "Formatted at 2026-09-28 04:43:40 on dist-test-slave-j5c3"
server_key: "f986926288ec79a012b012a8e6c98a80"
server_key_iv: "5bbc485cf19ae895eac5f02befe10838"
server_key_version: "encryptionkey@0"
I20260928 04:43:40.325855 9403 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:43:40.326583 9418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:40.326790 9403 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:43:40.326882 9403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "7c4ba4735141419a8026489a0785daa9"
format_stamp: "Formatted at 2026-09-28 04:43:40 on dist-test-slave-j5c3"
server_key: "f986926288ec79a012b012a8e6c98a80"
server_key_iv: "5bbc485cf19ae895eac5f02befe10838"
server_key_version: "encryptionkey@0"
I20260928 04:43:40.326939 9403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:40.343317 9403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:40.345481 9403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:40.345597 9403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:40.350108 9403 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:45703
I20260928 04:43:40.350211 9480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:45703 every 8 connection(s)
I20260928 04:43:40.350540 9403 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:43:40.351279 9481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:40.351998 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 9403
I20260928 04:43:40.352093 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:43:40.352316 1931 external_mini_cluster.cc:1428] Setting key d3acb848a2c6538a389a3882cce3a0aa
I20260928 04:43:40.354506 9481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c4ba4735141419a8026489a0785daa9: Bootstrap starting.
I20260928 04:43:40.355165 9481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c4ba4735141419a8026489a0785daa9: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:40.355449 9481 log.cc:824] T 00000000000000000000000000000000 P 7c4ba4735141419a8026489a0785daa9: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:40.356120 9481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c4ba4735141419a8026489a0785daa9: No bootstrap required, opened a new log
I20260928 04:43:40.358009 9481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c4ba4735141419a8026489a0785daa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4ba4735141419a8026489a0785daa9" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45703 } }
I20260928 04:43:40.358137 9481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c4ba4735141419a8026489a0785daa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:40.358160 9481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c4ba4735141419a8026489a0785daa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c4ba4735141419a8026489a0785daa9, State: Initialized, Role: FOLLOWER
I20260928 04:43:40.358299 9481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c4ba4735141419a8026489a0785daa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4ba4735141419a8026489a0785daa9" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45703 } }
I20260928 04:43:40.358376 9481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c4ba4735141419a8026489a0785daa9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:43:40.358397 9481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c4ba4735141419a8026489a0785daa9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:43:40.358470 9481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c4ba4735141419a8026489a0785daa9 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 04:43:40 dist-test-slave-j5c3 krb5kdc[9390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM
I20260928 04:43:40.359025 9481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c4ba4735141419a8026489a0785daa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4ba4735141419a8026489a0785daa9" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45703 } }
I20260928 04:43:40.359150 9481 leader_election.cc:302] T 00000000000000000000000000000000 P 7c4ba4735141419a8026489a0785daa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c4ba4735141419a8026489a0785daa9; no voters:
I20260928 04:43:40.359336 9481 leader_election.cc:288] T 00000000000000000000000000000000 P 7c4ba4735141419a8026489a0785daa9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:43:40.359364 9486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c4ba4735141419a8026489a0785daa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:40.359473 9486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c4ba4735141419a8026489a0785daa9 [term 1 LEADER]: Becoming Leader. State: Replica: 7c4ba4735141419a8026489a0785daa9, State: Running, Role: LEADER
I20260928 04:43:40.359563 9486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c4ba4735141419a8026489a0785daa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4ba4735141419a8026489a0785daa9" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45703 } }
I20260928 04:43:40.359594 9481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c4ba4735141419a8026489a0785daa9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:43:40.359892 9486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c4ba4735141419a8026489a0785daa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c4ba4735141419a8026489a0785daa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4ba4735141419a8026489a0785daa9" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45703 } } }
I20260928 04:43:40.360005 9486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c4ba4735141419a8026489a0785daa9 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:40.360174 9486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c4ba4735141419a8026489a0785daa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c4ba4735141419a8026489a0785daa9. Latest consensus state: current_term: 1 leader_uuid: "7c4ba4735141419a8026489a0785daa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4ba4735141419a8026489a0785daa9" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45703 } } }
I20260928 04:43:40.360250 9486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c4ba4735141419a8026489a0785daa9 [sys.catalog]: This master's current role is: LEADER
W20260928 04:43:40.361469 9502 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7c4ba4735141419a8026489a0785daa9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:43:40.361559 9502 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:43:40.361647 9492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:43:40.361873 9492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:43:40.363484 9492 catalog_manager.cc:1347] Generated new cluster ID: d25aa5e0d56645e98c20895d6bb4d14a
I20260928 04:43:40.363559 9492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:43:40.363482 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:40.352921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55520 (local address 127.1.226.254:45703)
0928 04:43:40.353080 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:40.353088 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:40.353114 (+ 26us) server_negotiation.cc:407] Connection header received
0928 04:43:40.353526 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:40.353533 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:40.353577 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:40.353689 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:40.354484 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.355605 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.356443 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.356884 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.357014 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:40.359554 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:40.359559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:40.359559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:40.359571 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:40.361817 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:40.362210 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:40.362214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:40.362214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:40.362323 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:40.362530 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:40.362531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:40.362531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:40.362833 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:40.362983 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:40.363076 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 04:43:40.363231 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260928 04:43:40.394491 9492 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:43:40.395253 9492 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:43:40.403164 9492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c4ba4735141419a8026489a0785daa9: Generated new TSK 0
I20260928 04:43:40.403388 9492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.226.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.226.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.226.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.193.keytab.
Entry for principal oryx/127.1.226.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.193.keytab.
I20260928 04:43:40.431797 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:0
--local_ip_for_outbound_sockets=127.1.226.193
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:45703
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.193.keytab
--principal=oryx/127.1.226.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:40.514557 9507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:40.514780 9507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:40.514806 9507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:40.516049 9507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:40.516112 9507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:40.516177 9507 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 04:43:40.516218 9507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:43:40.517827 9507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.193.keytab
--principal=oryx/127.1.226.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.226.254:45703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:40.518093 9507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:40.518347 9507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:40.523728 9513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:40.523803 9507 server_base.cc:1034] running on GCE node
W20260928 04:43:40.523741 9512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:40.523764 9515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:40.524253 9507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:40.524531 9507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:40.525743 9507 hybrid_clock.cc:648] HybridClock initialized: now 1790570620525733 us; error 32 us; skew 500 ppm
Sep 28 04:43:40 dist-test-slave-j5c3 krb5kdc[9390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570620, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:40.528385 9507 init.cc:377] Logged in from keytab as oryx/127.1.226.193@KRBTEST.COM (short username oryx)
I20260928 04:43:40.528813 9507 webserver.cc:460] Webserver started at http://127.1.226.193:34721/ using document root <none> and password file <none>
I20260928 04:43:40.529047 9507 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:40.529114 9507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:40.529268 9507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:40.530362 9507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-0/data/instance:
uuid: "fd16e3dcebe3400b8e6985dd10659cf5"
format_stamp: "Formatted at 2026-09-28 04:43:40 on dist-test-slave-j5c3"
server_key: "48e9aa6a0273577fe49cce778596cbb2"
server_key_iv: "3a31576e411ddc711036f988d3684d55"
server_key_version: "encryptionkey@0"
I20260928 04:43:40.530721 9507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance:
uuid: "fd16e3dcebe3400b8e6985dd10659cf5"
format_stamp: "Formatted at 2026-09-28 04:43:40 on dist-test-slave-j5c3"
server_key: "48e9aa6a0273577fe49cce778596cbb2"
server_key_iv: "3a31576e411ddc711036f988d3684d55"
server_key_version: "encryptionkey@0"
I20260928 04:43:40.532018 9507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:43:40.532790 9522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:40.532963 9507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:40.533061 9507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "fd16e3dcebe3400b8e6985dd10659cf5"
format_stamp: "Formatted at 2026-09-28 04:43:40 on dist-test-slave-j5c3"
server_key: "48e9aa6a0273577fe49cce778596cbb2"
server_key_iv: "3a31576e411ddc711036f988d3684d55"
server_key_version: "encryptionkey@0"
I20260928 04:43:40.533144 9507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:40.549633 9507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:40.551824 9507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:40.552031 9507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:40.552698 9507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:40.552775 9507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:40.552816 9507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:40.552865 9507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 04:43:40 dist-test-slave-j5c3 krb5kdc[9390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570620, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM
I20260928 04:43:40.561043 9507 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:40603
I20260928 04:43:40.561100 9637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:40603 every 8 connection(s)
I20260928 04:43:40.561519 9507 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
I20260928 04:43:40.563509 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:40.553218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:49357 (local address 127.1.226.254:45703)
0928 04:43:40.553296 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:40.553298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:40.553465 (+ 167us) server_negotiation.cc:407] Connection header received
0928 04:43:40.553977 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:40.553979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:40.553993 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:40.554092 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:40.555022 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.555846 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.557186 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.557404 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.557483 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:40.560310 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:40.560311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:40.560312 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:40.560320 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:40.562068 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:40.562515 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:40.562516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:40.562517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:40.562577 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:40.562849 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:40.562849 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:40.562850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:40.562972 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:40.563046 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:40.563332 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 04:43:40.563416 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 04:43:40.563702 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:40.553173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:45703 (local address 127.1.226.193:49357)
0928 04:43:40.553330 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:40.553372 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:40.553872 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:40.554212 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:40.554217 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:40.554602 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:40.554896 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:40.554905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.555965 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.555965 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:40.557074 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:40.557077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.557503 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.557504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:40.557637 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:40.558267 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:40.558278 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:40.560217 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:40.562168 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:40.562174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:40.562176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:40.562435 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:40.562652 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:40.562652 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:40.562653 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:40.562774 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:40.563039 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:40.563045 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:40.563195 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 04:43:40.563310 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 04:43:40.563475 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260928 04:43:40.564224 9638 heartbeater.cc:347] Connected to a master server at 127.1.226.254:45703
I20260928 04:43:40.564347 9638 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:40.564541 9638 heartbeater.cc:511] Master 127.1.226.254:45703 requested a full tablet report, sending...
I20260928 04:43:40.565131 9434 ts_manager.cc:194] Registered new tserver with Master: fd16e3dcebe3400b8e6985dd10659cf5 (127.1.226.193:40603)
I20260928 04:43:40.566030 9434 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.226.193@KRBTEST.COM'} at 127.1.226.193:49357
I20260928 04:43:40.566927 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 9507
I20260928 04:43:40.567018 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance
I20260928 04:43:40.567173 1931 external_mini_cluster.cc:1428] Setting key 62c3804028597d55ceb6e45dafbce198
I20260928 04:43:40.571261 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:40.565350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:45703 (local address 127.1.226.193:54013)
0928 04:43:40.565472 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:40.565482 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:40.565535 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:40.565659 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:40.565659 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:40.565815 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:40.565870 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:40.565871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.566652 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.566653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:40.567380 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:40.567382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.567731 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.567732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:40.567827 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:40.568319 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:40.568323 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:40.568614 (+ 291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:40.570158 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:40.570159 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:40.570159 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:40.570395 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:40.570604 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:40.570605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:40.570605 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:40.570663 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:40.570927 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:40.570928 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:40.570975 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 04:43:40.571058 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 04:43:40.571163 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":70,"threads_started":1}
I20260928 04:43:40.571257 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:40.565348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:54013 (local address 127.1.226.254:45703)
0928 04:43:40.565388 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:40.565390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:40.565546 (+ 156us) server_negotiation.cc:407] Connection header received
0928 04:43:40.565586 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:40.565586 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:40.565597 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:40.565647 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:40.565932 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.566556 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.567472 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.567663 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.567714 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:40.568693 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:40.568695 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:40.568695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:40.568702 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:40.570075 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:40.570473 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:40.570474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:40.570475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:40.570537 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:40.570735 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:40.570736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:40.570737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:40.570854 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:40.570924 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:40.571083 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 04:43:40.571160 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
WARNING: no policy specified for oryx/127.1.226.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.226.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.226.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.194.keytab.
Entry for principal oryx/127.1.226.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.194.keytab.
I20260928 04:43:40.592973 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:0
--local_ip_for_outbound_sockets=127.1.226.194
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:45703
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.194.keytab
--principal=oryx/127.1.226.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:40.674460 9647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:40.674729 9647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:40.674772 9647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:40.675969 9647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:40.676038 9647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:40.676075 9647 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 04:43:40.676115 9647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:43:40.677670 9647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.194.keytab
--principal=oryx/127.1.226.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.226.254:45703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:40.677943 9647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:40.678181 9647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:40.683164 9655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:40.683193 9653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:40.683166 9652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:40.683226 9647 server_base.cc:1034] running on GCE node
I20260928 04:43:40.683533 9647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:40.683766 9647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:40.684938 9647 hybrid_clock.cc:648] HybridClock initialized: now 1790570620684920 us; error 38 us; skew 500 ppm
Sep 28 04:43:40 dist-test-slave-j5c3 krb5kdc[9390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570620, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:40.687415 9647 init.cc:377] Logged in from keytab as oryx/127.1.226.194@KRBTEST.COM (short username oryx)
I20260928 04:43:40.687947 9647 webserver.cc:460] Webserver started at http://127.1.226.194:40041/ using document root <none> and password file <none>
I20260928 04:43:40.688223 9647 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:40.688292 9647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:40.688427 9647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:40.689461 9647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-1/data/instance:
uuid: "27aac7e335b4450da720babd10ed7809"
format_stamp: "Formatted at 2026-09-28 04:43:40 on dist-test-slave-j5c3"
server_key: "e2b5cca69d8fcd1e93e5a0ddeb88cc76"
server_key_iv: "6f1d6d26426df7c98e462f5fdf33e8a9"
server_key_version: "encryptionkey@0"
I20260928 04:43:40.689828 9647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance:
uuid: "27aac7e335b4450da720babd10ed7809"
format_stamp: "Formatted at 2026-09-28 04:43:40 on dist-test-slave-j5c3"
server_key: "e2b5cca69d8fcd1e93e5a0ddeb88cc76"
server_key_iv: "6f1d6d26426df7c98e462f5fdf33e8a9"
server_key_version: "encryptionkey@0"
I20260928 04:43:40.691128 9647 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:43:40.691846 9662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:40.692019 9647 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:43:40.692117 9647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "27aac7e335b4450da720babd10ed7809"
format_stamp: "Formatted at 2026-09-28 04:43:40 on dist-test-slave-j5c3"
server_key: "e2b5cca69d8fcd1e93e5a0ddeb88cc76"
server_key_iv: "6f1d6d26426df7c98e462f5fdf33e8a9"
server_key_version: "encryptionkey@0"
I20260928 04:43:40.692233 9647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:40.714115 9647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:40.716585 9647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:40.716753 9647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:40.717327 9647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:40.717386 9647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:40.717425 9647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:40.717447 9647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 04:43:40 dist-test-slave-j5c3 krb5kdc[9390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570620, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM
I20260928 04:43:40.725106 9647 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:44519
I20260928 04:43:40.725260 9777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:44519 every 8 connection(s)
I20260928 04:43:40.725536 9647 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
I20260928 04:43:40.728120 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 9647
I20260928 04:43:40.728276 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance
I20260928 04:43:40.728461 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:40.717723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:45703 (local address 127.1.226.194:40367)
0928 04:43:40.717915 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:40.717973 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:40.718725 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:40.719135 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:40.719140 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:40.719565 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:40.719940 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:40.719950 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.720778 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.720779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:40.722265 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:40.722268 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.722728 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.722730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:40.722871 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:40.723452 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:40.723460 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:40.725043 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:40.726852 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:40.726859 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:40.726861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:40.727089 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:40.727330 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:40.727331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:40.727331 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:40.727445 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:40.727711 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:40.727718 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:40.727865 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 04:43:40.727976 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 04:43:40.728194 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":99,"threads_started":1}
I20260928 04:43:40.728498 1931 external_mini_cluster.cc:1428] Setting key c89fe68cb7a5e734b9cf8af7c1a2e65c
I20260928 04:43:40.728914 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:40.717739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:40367 (local address 127.1.226.254:45703)
0928 04:43:40.717810 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:40.717812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:40.718075 (+ 263us) server_negotiation.cc:407] Connection header received
0928 04:43:40.718861 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:40.718864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:40.718880 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:40.718917 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:40.720041 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.720688 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.722379 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.722595 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.722702 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:40.725155 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:40.725156 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:40.725156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:40.725166 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:40.726724 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:40.727189 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:40.727191 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:40.727192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:40.727253 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:40.727511 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:40.727512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:40.727513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:40.727634 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:40.727722 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:40.728732 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0928 04:43:40.728817 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 04:43:40.729355 9778 heartbeater.cc:347] Connected to a master server at 127.1.226.254:45703
I20260928 04:43:40.729465 9778 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:40.729655 9778 heartbeater.cc:511] Master 127.1.226.254:45703 requested a full tablet report, sending...
I20260928 04:43:40.730123 9435 ts_manager.cc:194] Registered new tserver with Master: 27aac7e335b4450da720babd10ed7809 (127.1.226.194:44519)
I20260928 04:43:40.730679 9435 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.226.194@KRBTEST.COM'} at 127.1.226.194:40367
I20260928 04:43:40.735991 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:40.730425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:51537 (local address 127.1.226.254:45703)
0928 04:43:40.730465 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:40.730467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:40.730602 (+ 135us) server_negotiation.cc:407] Connection header received
0928 04:43:40.730635 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:40.730636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:40.730648 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:40.730705 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:40.731061 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.731512 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.732245 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.732452 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.732527 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:40.733506 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:40.733508 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:40.733508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:40.733515 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:40.734826 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:40.735164 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:40.735165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:40.735166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:40.735218 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:40.735429 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:40.735430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:40.735430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:40.735577 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:40.735657 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:40.735838 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 04:43:40.735908 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:43:40.735996 9784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:40.730363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:45703 (local address 127.1.226.194:51537)
0928 04:43:40.730522 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:40.730533 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:40.730595 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:40.730709 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:40.730709 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:40.730915 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:40.730975 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:40.730977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.731612 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.731613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:40.732130 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:40.732131 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.732526 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.732527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:40.732590 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:40.733063 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:40.733067 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:40.733424 (+ 357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:40.734908 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:40.734910 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:40.734910 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:40.735080 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:40.735307 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:40.735308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:40.735308 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:40.735369 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:40.735648 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:40.735649 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:40.735714 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 04:43:40.735776 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 04:43:40.735885 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for oryx/127.1.226.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.226.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.226.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.195.keytab.
Entry for principal oryx/127.1.226.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.195.keytab.
I20260928 04:43:40.756615 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:0
--local_ip_for_outbound_sockets=127.1.226.195
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:45703
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.195.keytab
--principal=oryx/127.1.226.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:40.850666 9787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:40.851011 9787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:40.851055 9787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:40.852427 9787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:40.852499 9787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:40.852536 9787 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 04:43:40.852588 9787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:43:40.854250 9787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.195.keytab
--principal=oryx/127.1.226.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.226.254:45703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:40.854547 9787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:40.854801 9787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:40.860065 9795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:40.860105 9793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:40.860105 9792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:40.860457 9787 server_base.cc:1034] running on GCE node
I20260928 04:43:40.860714 9787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:40.861012 9787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:40.862205 9787 hybrid_clock.cc:648] HybridClock initialized: now 1790570620862192 us; error 54 us; skew 500 ppm
Sep 28 04:43:40 dist-test-slave-j5c3 krb5kdc[9390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570620, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:40.864974 9787 init.cc:377] Logged in from keytab as oryx/127.1.226.195@KRBTEST.COM (short username oryx)
I20260928 04:43:40.865435 9787 webserver.cc:460] Webserver started at http://127.1.226.195:44351/ using document root <none> and password file <none>
I20260928 04:43:40.865661 9787 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:40.865705 9787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:40.865846 9787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:40.866911 9787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-2/data/instance:
uuid: "e867826874874526b87f5fef3a61a238"
format_stamp: "Formatted at 2026-09-28 04:43:40 on dist-test-slave-j5c3"
server_key: "791946a15eb0e708605dfc0cd8264055"
server_key_iv: "5826e3715a21ad7475d15ddbf7212bed"
server_key_version: "encryptionkey@0"
I20260928 04:43:40.867268 9787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance:
uuid: "e867826874874526b87f5fef3a61a238"
format_stamp: "Formatted at 2026-09-28 04:43:40 on dist-test-slave-j5c3"
server_key: "791946a15eb0e708605dfc0cd8264055"
server_key_iv: "5826e3715a21ad7475d15ddbf7212bed"
server_key_version: "encryptionkey@0"
I20260928 04:43:40.868722 9787 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:43:40.869582 9802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:40.869858 9787 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:43:40.869976 9787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "e867826874874526b87f5fef3a61a238"
format_stamp: "Formatted at 2026-09-28 04:43:40 on dist-test-slave-j5c3"
server_key: "791946a15eb0e708605dfc0cd8264055"
server_key_iv: "5826e3715a21ad7475d15ddbf7212bed"
server_key_version: "encryptionkey@0"
I20260928 04:43:40.870033 9787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:40.878178 9787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:40.880038 9787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:40.880211 9787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:40.880685 9787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:40.880720 9787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:40.880743 9787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:40.880755 9787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:40.887712 9787 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:34427
I20260928 04:43:40.887763 9917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:34427 every 8 connection(s)
I20260928 04:43:40.888144 9787 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
Sep 28 04:43:40 dist-test-slave-j5c3 krb5kdc[9390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570620, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM
I20260928 04:43:40.891633 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:40.881345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:57055 (local address 127.1.226.254:45703)
0928 04:43:40.881414 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:40.881416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:40.881579 (+ 163us) server_negotiation.cc:407] Connection header received
0928 04:43:40.882079 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:40.882080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:40.882096 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:40.882176 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:40.883115 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.883991 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.885651 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.885953 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.886043 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:40.888963 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:40.888965 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:40.888965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:40.888973 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:40.890489 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:40.890890 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:40.890891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:40.890891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:40.890946 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:40.891168 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:40.891168 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:40.891169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:40.891273 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:40.891324 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:40.891513 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 04:43:40.891570 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 04:43:40.891963 9811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:40.881280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:45703 (local address 127.1.226.195:57055)
0928 04:43:40.881453 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:40.881493 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:40.881980 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:40.882276 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:40.882298 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:40.882695 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:40.883003 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:40.883009 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.884092 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.884093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:40.885538 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:40.885542 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.886045 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.886045 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:40.886158 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:40.886895 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:40.886904 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:40.888879 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:40.890570 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:40.890575 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:40.890577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:40.890794 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:40.891006 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:40.891007 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:40.891007 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:40.891097 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:40.891316 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:40.891319 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:40.891425 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 04:43:40.891502 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 04:43:40.891722 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
I20260928 04:43:40.892246 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 9787
I20260928 04:43:40.892375 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance
I20260928 04:43:40.892630 1931 external_mini_cluster.cc:1428] Setting key 53336c8b749acd224a77d626f20c6a7f
I20260928 04:43:40.893463 9918 heartbeater.cc:347] Connected to a master server at 127.1.226.254:45703
I20260928 04:43:40.893549 9918 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:40.893731 9918 heartbeater.cc:511] Master 127.1.226.254:45703 requested a full tablet report, sending...
I20260928 04:43:40.894093 9435 ts_manager.cc:194] Registered new tserver with Master: e867826874874526b87f5fef3a61a238 (127.1.226.195:34427)
I20260928 04:43:40.894610 9435 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.226.195@KRBTEST.COM'} at 127.1.226.195:57055
I20260928 04:43:40.899101 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:40.893621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:53091 (local address 127.1.226.254:45703)
0928 04:43:40.893660 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:40.893662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:40.893822 (+ 160us) server_negotiation.cc:407] Connection header received
0928 04:43:40.893847 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:40.893848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:40.893858 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:40.893908 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:40.894234 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.894918 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.895681 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.895907 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.895973 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:40.896874 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:40.896875 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:40.896875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:40.896881 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:40.898270 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:40.898557 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:40.898558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:40.898558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:40.898598 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:40.898734 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:40.898734 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:40.898734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:40.898827 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:40.898872 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:40.898986 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 04:43:40.899037 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:43:40.899115 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:40.893557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:45703 (local address 127.1.226.195:53091)
0928 04:43:40.893724 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:40.893733 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:40.893783 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:40.893916 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:40.893917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:40.894064 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:40.894120 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:40.894121 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.895011 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.895012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:40.895613 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:40.895615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.895987 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.895987 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:40.896035 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:40.896483 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:40.896487 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:40.896786 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:40.898345 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:40.898345 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:40.898346 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:40.898501 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:40.898637 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:40.898637 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:40.898637 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:40.898688 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:40.898879 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:40.898879 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:40.898914 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 04:43:40.898970 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 04:43:40.899043 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":80,"threads_started":1}
I20260928 04:43:40.904171 1931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 04:43:40 dist-test-slave-j5c3 krb5kdc[9390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570620, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 04:43:40 dist-test-slave-j5c3 krb5kdc[9390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:40 dist-test-slave-j5c3 krb5kdc[9390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 04:43:40.926201 9484 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:40.920237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55532 (local address 127.1.226.254:45703)
0928 04:43:40.920310 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:40.920312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:40.920322 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:43:40.920355 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:40.920356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:40.920370 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:40.920465 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:40.921158 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.921717 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.922542 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.922746 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.922844 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:40.926079 (+ 3235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55532: BlockingRecv error: recv got EOF from 127.0.0.1:55532 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 04:43:40 dist-test-slave-j5c3 krb5kdc[9390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570620, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM
I20260928 04:43:40.934886 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:40.927081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55536 (local address 127.1.226.254:45703)
0928 04:43:40.927134 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:40.927136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:40.927147 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:43:40.927213 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:40.927214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:40.927223 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:40.927270 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:40.927724 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.928304 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.928990 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.929291 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.929454 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:40.932001 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:40.932003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:40.932003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:40.932012 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:40.933453 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:40.933943 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:40.933945 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:40.933946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:40.934018 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:40.934247 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:40.934248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:40.934248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:40.934380 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:40.934495 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:40.934665 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 04:43:40.934759 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 04:43:40.936818 9435 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:43:40.938493 9435 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:43:40.945037 9940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:40.940979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:44519 (local address 127.0.0.1:41202)
0928 04:43:40.941778 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:40.941797 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:40.941878 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:40.942143 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:40.942144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:40.942145 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:40.942191 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:40.942192 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.943026 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.943027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:40.943996 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:40.943998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.944816 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.944817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:40.944871 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:40.944883 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 04:43:40.944941 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 04:43:40.944979 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"thread_start_us":102,"threads_started":1}
I20260928 04:43:40.945341 9944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:40.941247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41202 (local address 127.1.226.194:44519)
0928 04:43:40.941419 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:40.941423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:40.941850 (+ 427us) server_negotiation.cc:407] Connection header received
0928 04:43:40.941968 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:40.941971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:40.941990 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:40.942071 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:40.942274 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.942941 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.944092 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.944724 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.944811 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:40.944885 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:40.944955 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 04:43:40.944992 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":88,"threads_started":1}
I20260928 04:43:40.945906 9943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:40.941147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50672 (local address 127.1.226.195:34427)
0928 04:43:40.941354 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:40.941360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:40.941541 (+ 181us) server_negotiation.cc:407] Connection header received
0928 04:43:40.941589 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:40.941591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:40.941607 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:40.941693 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:40.941902 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.943140 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.944567 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.945298 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.945375 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:40.945454 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:40.945731 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 04:43:40.945789 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":101,"threads_started":1}
I20260928 04:43:40.946219 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:40.940975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:34427 (local address 127.0.0.1:50672)
0928 04:43:40.941430 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:40.941440 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:40.941496 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:40.941722 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:40.941725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:40.941727 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:40.941804 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:40.941808 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.943254 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.943255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:40.944479 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:40.944480 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.945427 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.945428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:40.945507 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:40.945512 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:43:40.945999 (+ 487us) client_negotiation.cc:240] Negotiation successful
0928 04:43:40.946105 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":71,"spinlock_wait_cycles":384,"thread_start_us":104,"threads_started":1}
I20260928 04:43:40.946156 9712 tablet_service.cc:1467] Processing CreateTablet for tablet 5869c1cfd2244da5b03118259f963e47 (DEFAULT_TABLE table=test-table [id=d4abc994aa324d809d81eff9e74feb43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:40.946630 9712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5869c1cfd2244da5b03118259f963e47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:40.946723 9945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:40.941258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38978 (local address 127.1.226.193:40603)
0928 04:43:40.941535 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:40.941539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:40.941550 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:43:40.941589 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:40.941591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:40.941607 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:40.941693 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:40.942062 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.943160 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.945494 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.946224 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.946298 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:40.946375 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:40.946567 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 04:43:40.946619 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":88,"threads_started":1}
I20260928 04:43:40.946964 9942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:40.940972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:40603 (local address 127.0.0.1:38978)
0928 04:43:40.941182 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:40.941228 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:40.941301 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:40.941767 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:40.941767 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:40.941769 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:40.941804 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:40.941806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.943254 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.943255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:40.944477 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:40.944480 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:40.946410 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:40.946411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:40.946471 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:40.946474 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:43:40.946870 (+ 396us) client_negotiation.cc:240] Negotiation successful
0928 04:43:40.946912 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":42,"thread_start_us":62,"threads_started":1}
I20260928 04:43:40.946982 9852 tablet_service.cc:1467] Processing CreateTablet for tablet 5869c1cfd2244da5b03118259f963e47 (DEFAULT_TABLE table=test-table [id=d4abc994aa324d809d81eff9e74feb43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:40.947360 9852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5869c1cfd2244da5b03118259f963e47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:40.947576 9572 tablet_service.cc:1467] Processing CreateTablet for tablet 5869c1cfd2244da5b03118259f963e47 (DEFAULT_TABLE table=test-table [id=d4abc994aa324d809d81eff9e74feb43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:40.947858 9572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5869c1cfd2244da5b03118259f963e47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:40.951172 9946 tablet_bootstrap.cc:492] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809: Bootstrap starting.
I20260928 04:43:40.951889 9946 tablet_bootstrap.cc:654] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:40.952243 9946 log.cc:824] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:40.952327 9947 tablet_bootstrap.cc:492] T 5869c1cfd2244da5b03118259f963e47 P fd16e3dcebe3400b8e6985dd10659cf5: Bootstrap starting.
I20260928 04:43:40.952373 9948 tablet_bootstrap.cc:492] T 5869c1cfd2244da5b03118259f963e47 P e867826874874526b87f5fef3a61a238: Bootstrap starting.
I20260928 04:43:40.952944 9947 tablet_bootstrap.cc:654] T 5869c1cfd2244da5b03118259f963e47 P fd16e3dcebe3400b8e6985dd10659cf5: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:40.953022 9948 tablet_bootstrap.cc:654] T 5869c1cfd2244da5b03118259f963e47 P e867826874874526b87f5fef3a61a238: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:40.953239 9946 tablet_bootstrap.cc:492] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809: No bootstrap required, opened a new log
I20260928 04:43:40.953253 9947 log.cc:824] T 5869c1cfd2244da5b03118259f963e47 P fd16e3dcebe3400b8e6985dd10659cf5: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:40.953316 9946 ts_tablet_manager.cc:1397] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 04:43:40.953352 9948 log.cc:824] T 5869c1cfd2244da5b03118259f963e47 P e867826874874526b87f5fef3a61a238: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:40.954061 9947 tablet_bootstrap.cc:492] T 5869c1cfd2244da5b03118259f963e47 P fd16e3dcebe3400b8e6985dd10659cf5: No bootstrap required, opened a new log
I20260928 04:43:40.954082 9948 tablet_bootstrap.cc:492] T 5869c1cfd2244da5b03118259f963e47 P e867826874874526b87f5fef3a61a238: No bootstrap required, opened a new log
I20260928 04:43:40.954136 9948 ts_tablet_manager.cc:1397] T 5869c1cfd2244da5b03118259f963e47 P e867826874874526b87f5fef3a61a238: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:43:40.954136 9947 ts_tablet_manager.cc:1397] T 5869c1cfd2244da5b03118259f963e47 P fd16e3dcebe3400b8e6985dd10659cf5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:43:40.954998 9946 raft_consensus.cc:348] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:40.955156 9946 raft_consensus.cc:374] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:40.955200 9946 raft_consensus.cc:729] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aac7e335b4450da720babd10ed7809, State: Initialized, Role: FOLLOWER
I20260928 04:43:40.955314 9946 consensus_queue.cc:260] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:40.955615 9946 ts_tablet_manager.cc:1428] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 04:43:40.955619 9778 heartbeater.cc:503] Master 127.1.226.254:45703 was elected leader, sending a full tablet report...
I20260928 04:43:40.956132 9947 raft_consensus.cc:348] T 5869c1cfd2244da5b03118259f963e47 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:40.956132 9948 raft_consensus.cc:348] T 5869c1cfd2244da5b03118259f963e47 P e867826874874526b87f5fef3a61a238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:40.956338 9947 raft_consensus.cc:374] T 5869c1cfd2244da5b03118259f963e47 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:40.956339 9948 raft_consensus.cc:374] T 5869c1cfd2244da5b03118259f963e47 P e867826874874526b87f5fef3a61a238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:40.956375 9948 raft_consensus.cc:729] T 5869c1cfd2244da5b03118259f963e47 P e867826874874526b87f5fef3a61a238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e867826874874526b87f5fef3a61a238, State: Initialized, Role: FOLLOWER
I20260928 04:43:40.956375 9947 raft_consensus.cc:729] T 5869c1cfd2244da5b03118259f963e47 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd16e3dcebe3400b8e6985dd10659cf5, State: Initialized, Role: FOLLOWER
I20260928 04:43:40.956496 9947 consensus_queue.cc:260] T 5869c1cfd2244da5b03118259f963e47 P fd16e3dcebe3400b8e6985dd10659cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:40.956496 9948 consensus_queue.cc:260] T 5869c1cfd2244da5b03118259f963e47 P e867826874874526b87f5fef3a61a238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:40.956768 9947 ts_tablet_manager.cc:1428] T 5869c1cfd2244da5b03118259f963e47 P fd16e3dcebe3400b8e6985dd10659cf5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 04:43:40.956768 9948 ts_tablet_manager.cc:1428] T 5869c1cfd2244da5b03118259f963e47 P e867826874874526b87f5fef3a61a238: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 04:43:40.956817 9638 heartbeater.cc:503] Master 127.1.226.254:45703 was elected leader, sending a full tablet report...
I20260928 04:43:40.957089 9918 heartbeater.cc:503] Master 127.1.226.254:45703 was elected leader, sending a full tablet report...
W20260928 04:43:40.976106 9779 tablet.cc:2369] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:43:41.062350 9639 tablet.cc:2369] T 5869c1cfd2244da5b03118259f963e47 P fd16e3dcebe3400b8e6985dd10659cf5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:43:41.138938 9919 tablet.cc:2369] T 5869c1cfd2244da5b03118259f963e47 P e867826874874526b87f5fef3a61a238: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:43:41.158958 9952 raft_consensus.cc:482] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:43:41.159133 9952 raft_consensus.cc:504] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.159521 9952 leader_election.cc:288] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd16e3dcebe3400b8e6985dd10659cf5 (127.1.226.193:40603), e867826874874526b87f5fef3a61a238 (127.1.226.195:34427)
I20260928 04:43:41.161002 9953 raft_consensus.cc:482] T 5869c1cfd2244da5b03118259f963e47 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:43:41.161132 9953 raft_consensus.cc:504] T 5869c1cfd2244da5b03118259f963e47 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.161870 9953 leader_election.cc:288] T 5869c1cfd2244da5b03118259f963e47 P fd16e3dcebe3400b8e6985dd10659cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aac7e335b4450da720babd10ed7809 (127.1.226.194:44519), e867826874874526b87f5fef3a61a238 (127.1.226.195:34427)
I20260928 04:43:41.163115 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:41.159737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:40603 (local address 127.1.226.194:42227)
0928 04:43:41.159815 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:41.159829 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:41.159889 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:41.160061 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:41.160062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:41.160064 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:41.160135 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:41.160137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.160875 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.160876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:41.162083 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:41.162085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.162831 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.162831 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:41.162927 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:41.162930 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:43:41.162977 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 04:43:41.163023 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 04:43:41.163347 9943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:41.159914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:50887 (local address 127.1.226.195:34427)
0928 04:43:41.159967 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:41.159969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:41.160082 (+ 113us) server_negotiation.cc:407] Connection header received
0928 04:43:41.160116 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:41.160117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:41.160127 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:41.160254 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:41.160427 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.161037 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.162448 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.162979 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.163017 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:41.163024 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:41.163226 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 04:43:41.163274 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 04:43:41.163376 9945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:41.159770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:42227 (local address 127.1.226.193:40603)
0928 04:43:41.159843 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:41.159845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:41.159886 (+ 41us) server_negotiation.cc:407] Connection header received
0928 04:43:41.159969 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:41.159971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:41.159981 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:41.160042 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:41.160240 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.160794 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.162183 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.162721 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.162782 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:41.162802 (+ 20us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:41.163279 (+ 477us) server_negotiation.cc:299] Negotiation successful
0928 04:43:41.163320 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 04:43:41.163550 9955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:41.159743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:34427 (local address 127.1.226.194:50887)
0928 04:43:41.160004 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:41.160016 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:41.160070 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:41.160257 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:41.160259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:41.160273 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:41.160354 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:41.160355 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.161152 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.161153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:41.162362 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:41.162364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.163077 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.163077 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:41.163140 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:41.163144 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:43:41.163413 (+ 269us) client_negotiation.cc:240] Negotiation successful
0928 04:43:41.163457 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":115,"thread_start_us":96,"threads_started":1}
I20260928 04:43:41.163604 9592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5869c1cfd2244da5b03118259f963e47" candidate_uuid: "27aac7e335b4450da720babd10ed7809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" is_pre_election: true
I20260928 04:43:41.163700 9592 raft_consensus.cc:2454] T 5869c1cfd2244da5b03118259f963e47 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aac7e335b4450da720babd10ed7809 in term 0.
I20260928 04:43:41.163949 9663 leader_election.cc:302] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aac7e335b4450da720babd10ed7809, fd16e3dcebe3400b8e6985dd10659cf5; no voters:
I20260928 04:43:41.163997 9872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5869c1cfd2244da5b03118259f963e47" candidate_uuid: "27aac7e335b4450da720babd10ed7809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e867826874874526b87f5fef3a61a238" is_pre_election: true
I20260928 04:43:41.164072 9952 raft_consensus.cc:2781] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:43:41.164110 9952 raft_consensus.cc:482] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:43:41.164121 9872 raft_consensus.cc:2454] T 5869c1cfd2244da5b03118259f963e47 P e867826874874526b87f5fef3a61a238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aac7e335b4450da720babd10ed7809 in term 0.
I20260928 04:43:41.164134 9952 raft_consensus.cc:3037] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:41.165120 9952 raft_consensus.cc:504] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.165251 9952 leader_election.cc:288] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809 [CANDIDATE]: Term 1 election: Requested vote from peers fd16e3dcebe3400b8e6985dd10659cf5 (127.1.226.193:40603), e867826874874526b87f5fef3a61a238 (127.1.226.195:34427)
I20260928 04:43:41.165465 9592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5869c1cfd2244da5b03118259f963e47" candidate_uuid: "27aac7e335b4450da720babd10ed7809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd16e3dcebe3400b8e6985dd10659cf5"
I20260928 04:43:41.165490 9872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5869c1cfd2244da5b03118259f963e47" candidate_uuid: "27aac7e335b4450da720babd10ed7809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e867826874874526b87f5fef3a61a238"
I20260928 04:43:41.165548 9592 raft_consensus.cc:3037] T 5869c1cfd2244da5b03118259f963e47 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:41.165598 9872 raft_consensus.cc:3037] T 5869c1cfd2244da5b03118259f963e47 P e867826874874526b87f5fef3a61a238 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:41.166235 9944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:41.161757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:57325 (local address 127.1.226.194:44519)
0928 04:43:41.161795 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:41.161797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:41.162031 (+ 234us) server_negotiation.cc:407] Connection header received
0928 04:43:41.162094 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:41.162095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:41.162105 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:41.162166 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:41.162353 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.163450 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.165358 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.165871 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.165952 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:41.165958 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:41.166135 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 04:43:41.166176 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:43:41.166234 9957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:41.161610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:44519 (local address 127.1.226.193:57325)
0928 04:43:41.161977 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:41.161987 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:41.162046 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:41.162211 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:41.162211 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:41.162214 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:41.162272 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:41.162274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.164326 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.164327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:41.165288 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:41.165291 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.165970 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.165972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:41.166048 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:41.166054 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 04:43:41.166114 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 04:43:41.166155 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":67,"threads_started":1}
I20260928 04:43:41.166431 9592 raft_consensus.cc:2454] T 5869c1cfd2244da5b03118259f963e47 P fd16e3dcebe3400b8e6985dd10659cf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aac7e335b4450da720babd10ed7809 in term 1.
I20260928 04:43:41.166687 9663 leader_election.cc:302] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aac7e335b4450da720babd10ed7809, fd16e3dcebe3400b8e6985dd10659cf5; no voters:
I20260928 04:43:41.166764 9872 raft_consensus.cc:2454] T 5869c1cfd2244da5b03118259f963e47 P e867826874874526b87f5fef3a61a238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aac7e335b4450da720babd10ed7809 in term 1.
I20260928 04:43:41.166893 9956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:41.161610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:34427 (local address 127.1.226.193:54909)
0928 04:43:41.162122 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:41.162129 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:41.162170 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:41.162358 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:41.162359 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:41.162361 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:43:41.162417 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:41.162419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.163366 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.163367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:41.164717 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:41.164720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.166480 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.166481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:41.166534 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:41.166537 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:43:41.166761 (+ 224us) client_negotiation.cc:240] Negotiation successful
0928 04:43:41.166802 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":11,"thread_start_us":60,"threads_started":1}
I20260928 04:43:41.166864 9732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5869c1cfd2244da5b03118259f963e47" candidate_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aac7e335b4450da720babd10ed7809" is_pre_election: true
I20260928 04:43:41.166993 9732 raft_consensus.cc:2379] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fd16e3dcebe3400b8e6985dd10659cf5 in current term 1: Already voted for candidate 27aac7e335b4450da720babd10ed7809 in this term.
I20260928 04:43:41.167088 9958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:41.161762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:54909 (local address 127.1.226.195:34427)
0928 04:43:41.161894 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:41.161895 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:41.162220 (+ 325us) server_negotiation.cc:407] Connection header received
0928 04:43:41.162242 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:41.162243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:41.162263 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:41.162320 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:43:41.162488 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.163264 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.164805 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.165364 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.165432 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:41.165440 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:41.166965 (+ 1525us) server_negotiation.cc:299] Negotiation successful
0928 04:43:41.167008 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260928 04:43:41.167306 9952 raft_consensus.cc:2781] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:41.167464 9952 raft_consensus.cc:686] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809 [term 1 LEADER]: Becoming Leader. State: Replica: 27aac7e335b4450da720babd10ed7809, State: Running, Role: LEADER
I20260928 04:43:41.167577 9952 consensus_queue.cc:237] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.167995 9872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5869c1cfd2244da5b03118259f963e47" candidate_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e867826874874526b87f5fef3a61a238" is_pre_election: true
I20260928 04:43:41.168099 9872 raft_consensus.cc:2379] T 5869c1cfd2244da5b03118259f963e47 P e867826874874526b87f5fef3a61a238 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fd16e3dcebe3400b8e6985dd10659cf5 in current term 1: Already voted for candidate 27aac7e335b4450da720babd10ed7809 in this term.
I20260928 04:43:41.168352 9525 leader_election.cc:302] T 5869c1cfd2244da5b03118259f963e47 P fd16e3dcebe3400b8e6985dd10659cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd16e3dcebe3400b8e6985dd10659cf5; no voters: 27aac7e335b4450da720babd10ed7809, e867826874874526b87f5fef3a61a238
I20260928 04:43:41.168385 9435 catalog_manager.cc:5495] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27aac7e335b4450da720babd10ed7809 (127.1.226.194). New cstate: current_term: 1 leader_uuid: "27aac7e335b4450da720babd10ed7809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } health_report { overall_health: UNKNOWN } } }
I20260928 04:43:41.168545 9953 raft_consensus.cc:2726] T 5869c1cfd2244da5b03118259f963e47 P fd16e3dcebe3400b8e6985dd10659cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 04:43:41.228830 9944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:41.226177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41212 (local address 127.1.226.194:44519)
0928 04:43:41.226226 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:41.226228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:41.226238 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:43:41.226301 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:41.226302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:41.226325 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:41.226398 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:43:41.226553 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.227036 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.227940 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.228213 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.228282 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:41.228473 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:43:41.228597 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:43:41.228697 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:41.228722 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 04:43:41.228744 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 04:43:41.230706 9872 raft_consensus.cc:1261] T 5869c1cfd2244da5b03118259f963e47 P e867826874874526b87f5fef3a61a238 [term 1 FOLLOWER]: Refusing update from remote peer 27aac7e335b4450da720babd10ed7809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:41.230749 9592 raft_consensus.cc:1261] T 5869c1cfd2244da5b03118259f963e47 P fd16e3dcebe3400b8e6985dd10659cf5 [term 1 FOLLOWER]: Refusing update from remote peer 27aac7e335b4450da720babd10ed7809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:41.231068 9959 consensus_queue.cc:1035] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:41.231207 9952 consensus_queue.cc:1035] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:41.233584 9963 mvcc.cc:205] Tried to move back new op lower bound from 7334177264558256128 to 7334177264302940160. Current Snapshot: MvccSnapshot[applied={T|T < 7334177264558256128}]
I20260928 04:43:41.233687 9964 mvcc.cc:205] Tried to move back new op lower bound from 7334177264558256128 to 7334177264302940160. Current Snapshot: MvccSnapshot[applied={T|T < 7334177264558256128}]
I20260928 04:43:41.235769 9434 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:43:41.235879 9434 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:43:41.236707 9434 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c4ba4735141419a8026489a0785daa9: Sending DeleteTablet for 3 replicas of tablet 5869c1cfd2244da5b03118259f963e47
I20260928 04:43:41.237026 9712 tablet_service.cc:1514] Processing DeleteTablet for tablet 5869c1cfd2244da5b03118259f963e47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:41 UTC) from {username='oryx', principal='oryx/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:41202
I20260928 04:43:41.237087 9852 tablet_service.cc:1514] Processing DeleteTablet for tablet 5869c1cfd2244da5b03118259f963e47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:41 UTC) from {username='oryx', principal='oryx/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:50672
I20260928 04:43:41.237075 9572 tablet_service.cc:1514] Processing DeleteTablet for tablet 5869c1cfd2244da5b03118259f963e47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:41 UTC) from {username='oryx', principal='oryx/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:38978
I20260928 04:43:41.237301 9972 tablet_replica.cc:330] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809: stopping tablet replica
I20260928 04:43:41.237404 9972 raft_consensus.cc:2229] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:43:41.237532 9972 raft_consensus.cc:2258] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:41.237582 9434 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:43:41.237730 9434 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:43:41.238042 9972 ts_tablet_manager.cc:1905] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:41.239341 9712 tablet_service.cc:1467] Processing CreateTablet for tablet 6a768c67cc5f436fa2efa34bd4217c26 (DEFAULT_TABLE table=test-table [id=8c2b96ba30ce41fdba7e24c34ad8915d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:41.239477 9712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a768c67cc5f436fa2efa34bd4217c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:41.239589 9572 tablet_service.cc:1467] Processing CreateTablet for tablet 6a768c67cc5f436fa2efa34bd4217c26 (DEFAULT_TABLE table=test-table [id=8c2b96ba30ce41fdba7e24c34ad8915d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:41.239707 9572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a768c67cc5f436fa2efa34bd4217c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:41.240044 9972 ts_tablet_manager.cc:1918] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:43:41.240113 9972 log.cc:1196] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-1/wal/wals/5869c1cfd2244da5b03118259f963e47
I20260928 04:43:41.240391 9972 ts_tablet_manager.cc:1939] T 5869c1cfd2244da5b03118259f963e47 P 27aac7e335b4450da720babd10ed7809: Deleting consensus metadata
I20260928 04:43:41.240621 9946 tablet_bootstrap.cc:492] T 6a768c67cc5f436fa2efa34bd4217c26 P 27aac7e335b4450da720babd10ed7809: Bootstrap starting.
I20260928 04:43:41.240777 9422 catalog_manager.cc:4841] TS 27aac7e335b4450da720babd10ed7809 (127.1.226.194:44519): tablet 5869c1cfd2244da5b03118259f963e47 (table test-table [id=d4abc994aa324d809d81eff9e74feb43]) successfully deleted
I20260928 04:43:41.241036 9946 tablet_bootstrap.cc:654] T 6a768c67cc5f436fa2efa34bd4217c26 P 27aac7e335b4450da720babd10ed7809: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:41.241073 9947 tablet_bootstrap.cc:492] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5: Bootstrap starting.
I20260928 04:43:41.241504 9947 tablet_bootstrap.cc:654] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:41.241492 9852 tablet_service.cc:1467] Processing CreateTablet for tablet 6a768c67cc5f436fa2efa34bd4217c26 (DEFAULT_TABLE table=test-table [id=8c2b96ba30ce41fdba7e24c34ad8915d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:41.241597 9852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a768c67cc5f436fa2efa34bd4217c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:41.242111 9946 tablet_bootstrap.cc:492] T 6a768c67cc5f436fa2efa34bd4217c26 P 27aac7e335b4450da720babd10ed7809: No bootstrap required, opened a new log
I20260928 04:43:41.242161 9946 ts_tablet_manager.cc:1397] T 6a768c67cc5f436fa2efa34bd4217c26 P 27aac7e335b4450da720babd10ed7809: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 04:43:41.242295 9946 raft_consensus.cc:348] T 6a768c67cc5f436fa2efa34bd4217c26 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.242386 9946 raft_consensus.cc:374] T 6a768c67cc5f436fa2efa34bd4217c26 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:41.242408 9946 raft_consensus.cc:729] T 6a768c67cc5f436fa2efa34bd4217c26 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aac7e335b4450da720babd10ed7809, State: Initialized, Role: FOLLOWER
I20260928 04:43:41.242470 9946 consensus_queue.cc:260] T 6a768c67cc5f436fa2efa34bd4217c26 P 27aac7e335b4450da720babd10ed7809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.242563 9946 ts_tablet_manager.cc:1428] T 6a768c67cc5f436fa2efa34bd4217c26 P 27aac7e335b4450da720babd10ed7809: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:41.242754 9947 tablet_bootstrap.cc:492] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5: No bootstrap required, opened a new log
I20260928 04:43:41.242815 9947 ts_tablet_manager.cc:1397] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 04:43:41.242962 9947 raft_consensus.cc:348] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.243034 9947 raft_consensus.cc:374] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:41.243028 9948 tablet_bootstrap.cc:492] T 6a768c67cc5f436fa2efa34bd4217c26 P e867826874874526b87f5fef3a61a238: Bootstrap starting.
I20260928 04:43:41.243057 9947 raft_consensus.cc:729] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd16e3dcebe3400b8e6985dd10659cf5, State: Initialized, Role: FOLLOWER
I20260928 04:43:41.243103 9947 consensus_queue.cc:260] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.238134 9974 tablet_replica.cc:330] T 5869c1cfd2244da5b03118259f963e47 P fd16e3dcebe3400b8e6985dd10659cf5: stopping tablet replica
I20260928 04:43:41.243193 9947 ts_tablet_manager.cc:1428] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 04:43:41.243299 9974 raft_consensus.cc:2229] T 5869c1cfd2244da5b03118259f963e47 P fd16e3dcebe3400b8e6985dd10659cf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:43:41.243407 9974 raft_consensus.cc:2258] T 5869c1cfd2244da5b03118259f963e47 P fd16e3dcebe3400b8e6985dd10659cf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:41.243428 9948 tablet_bootstrap.cc:654] T 6a768c67cc5f436fa2efa34bd4217c26 P e867826874874526b87f5fef3a61a238: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:41.243561 9973 tablet_replica.cc:330] T 5869c1cfd2244da5b03118259f963e47 P e867826874874526b87f5fef3a61a238: stopping tablet replica
I20260928 04:43:41.243655 9973 raft_consensus.cc:2229] T 5869c1cfd2244da5b03118259f963e47 P e867826874874526b87f5fef3a61a238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:43:41.244078 9974 ts_tablet_manager.cc:1905] T 5869c1cfd2244da5b03118259f963e47 P fd16e3dcebe3400b8e6985dd10659cf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:41.244376 9948 tablet_bootstrap.cc:492] T 6a768c67cc5f436fa2efa34bd4217c26 P e867826874874526b87f5fef3a61a238: No bootstrap required, opened a new log
I20260928 04:43:41.244423 9948 ts_tablet_manager.cc:1397] T 6a768c67cc5f436fa2efa34bd4217c26 P e867826874874526b87f5fef3a61a238: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 04:43:41.244557 9948 raft_consensus.cc:348] T 6a768c67cc5f436fa2efa34bd4217c26 P e867826874874526b87f5fef3a61a238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.244632 9948 raft_consensus.cc:374] T 6a768c67cc5f436fa2efa34bd4217c26 P e867826874874526b87f5fef3a61a238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:41.244653 9948 raft_consensus.cc:729] T 6a768c67cc5f436fa2efa34bd4217c26 P e867826874874526b87f5fef3a61a238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e867826874874526b87f5fef3a61a238, State: Initialized, Role: FOLLOWER
I20260928 04:43:41.244702 9948 consensus_queue.cc:260] T 6a768c67cc5f436fa2efa34bd4217c26 P e867826874874526b87f5fef3a61a238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.244778 9948 ts_tablet_manager.cc:1428] T 6a768c67cc5f436fa2efa34bd4217c26 P e867826874874526b87f5fef3a61a238: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:41.245627 9973 raft_consensus.cc:2258] T 5869c1cfd2244da5b03118259f963e47 P e867826874874526b87f5fef3a61a238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:41.245669 9974 ts_tablet_manager.cc:1918] T 5869c1cfd2244da5b03118259f963e47 P fd16e3dcebe3400b8e6985dd10659cf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:43:41.245728 9974 log.cc:1196] T 5869c1cfd2244da5b03118259f963e47 P fd16e3dcebe3400b8e6985dd10659cf5: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-0/wal/wals/5869c1cfd2244da5b03118259f963e47
I20260928 04:43:41.246012 9974 ts_tablet_manager.cc:1939] T 5869c1cfd2244da5b03118259f963e47 P fd16e3dcebe3400b8e6985dd10659cf5: Deleting consensus metadata
I20260928 04:43:41.246096 9973 ts_tablet_manager.cc:1905] T 5869c1cfd2244da5b03118259f963e47 P e867826874874526b87f5fef3a61a238: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:41.246425 9419 catalog_manager.cc:4841] TS fd16e3dcebe3400b8e6985dd10659cf5 (127.1.226.193:40603): tablet 5869c1cfd2244da5b03118259f963e47 (table test-table [id=d4abc994aa324d809d81eff9e74feb43]) successfully deleted
I20260928 04:43:41.247483 9973 ts_tablet_manager.cc:1918] T 5869c1cfd2244da5b03118259f963e47 P e867826874874526b87f5fef3a61a238: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:43:41.247546 9973 log.cc:1196] T 5869c1cfd2244da5b03118259f963e47 P e867826874874526b87f5fef3a61a238: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-2/wal/wals/5869c1cfd2244da5b03118259f963e47
I20260928 04:43:41.247780 9973 ts_tablet_manager.cc:1939] T 5869c1cfd2244da5b03118259f963e47 P e867826874874526b87f5fef3a61a238: Deleting consensus metadata
I20260928 04:43:41.248190 9421 catalog_manager.cc:4841] TS e867826874874526b87f5fef3a61a238 (127.1.226.195:34427): tablet 5869c1cfd2244da5b03118259f963e47 (table test-table [id=d4abc994aa324d809d81eff9e74feb43]) successfully deleted
I20260928 04:43:41.304636 9953 raft_consensus.cc:482] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:43:41.304811 9953 raft_consensus.cc:504] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.305006 9953 leader_election.cc:288] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aac7e335b4450da720babd10ed7809 (127.1.226.194:44519), e867826874874526b87f5fef3a61a238 (127.1.226.195:34427)
I20260928 04:43:41.305318 9732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a768c67cc5f436fa2efa34bd4217c26" candidate_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aac7e335b4450da720babd10ed7809" is_pre_election: true
I20260928 04:43:41.305425 9732 raft_consensus.cc:2454] T 6a768c67cc5f436fa2efa34bd4217c26 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd16e3dcebe3400b8e6985dd10659cf5 in term 0.
I20260928 04:43:41.305380 9872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a768c67cc5f436fa2efa34bd4217c26" candidate_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e867826874874526b87f5fef3a61a238" is_pre_election: true
I20260928 04:43:41.305511 9872 raft_consensus.cc:2454] T 6a768c67cc5f436fa2efa34bd4217c26 P e867826874874526b87f5fef3a61a238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd16e3dcebe3400b8e6985dd10659cf5 in term 0.
I20260928 04:43:41.305629 9526 leader_election.cc:302] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aac7e335b4450da720babd10ed7809, fd16e3dcebe3400b8e6985dd10659cf5; no voters:
I20260928 04:43:41.305734 9953 raft_consensus.cc:2781] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:43:41.305768 9953 raft_consensus.cc:482] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:43:41.305786 9953 raft_consensus.cc:3037] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:41.306483 9953 raft_consensus.cc:504] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.306623 9953 leader_election.cc:288] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5 [CANDIDATE]: Term 1 election: Requested vote from peers 27aac7e335b4450da720babd10ed7809 (127.1.226.194:44519), e867826874874526b87f5fef3a61a238 (127.1.226.195:34427)
I20260928 04:43:41.306779 9732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a768c67cc5f436fa2efa34bd4217c26" candidate_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aac7e335b4450da720babd10ed7809"
I20260928 04:43:41.306811 9872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a768c67cc5f436fa2efa34bd4217c26" candidate_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e867826874874526b87f5fef3a61a238"
I20260928 04:43:41.306864 9732 raft_consensus.cc:3037] T 6a768c67cc5f436fa2efa34bd4217c26 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:41.306869 9872 raft_consensus.cc:3037] T 6a768c67cc5f436fa2efa34bd4217c26 P e867826874874526b87f5fef3a61a238 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:41.307595 9872 raft_consensus.cc:2454] T 6a768c67cc5f436fa2efa34bd4217c26 P e867826874874526b87f5fef3a61a238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd16e3dcebe3400b8e6985dd10659cf5 in term 1.
I20260928 04:43:41.307595 9732 raft_consensus.cc:2454] T 6a768c67cc5f436fa2efa34bd4217c26 P 27aac7e335b4450da720babd10ed7809 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd16e3dcebe3400b8e6985dd10659cf5 in term 1.
I20260928 04:43:41.307761 9525 leader_election.cc:302] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e867826874874526b87f5fef3a61a238, fd16e3dcebe3400b8e6985dd10659cf5; no voters:
I20260928 04:43:41.307893 9953 raft_consensus.cc:2781] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:41.308017 9953 raft_consensus.cc:686] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5 [term 1 LEADER]: Becoming Leader. State: Replica: fd16e3dcebe3400b8e6985dd10659cf5, State: Running, Role: LEADER
I20260928 04:43:41.308102 9953 consensus_queue.cc:237] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.308924 9435 catalog_manager.cc:5495] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd16e3dcebe3400b8e6985dd10659cf5 (127.1.226.193). New cstate: current_term: 1 leader_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } health_report { overall_health: UNKNOWN } } }
Sep 28 04:43:41 dist-test-slave-j5c3 krb5kdc[9390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570620, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.254@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM
I20260928 04:43:41.344869 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:41.337077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55546 (local address 127.1.226.254:45703)
0928 04:43:41.337130 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:41.337132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:41.337268 (+ 136us) server_negotiation.cc:407] Connection header received
0928 04:43:41.337342 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:41.337342 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:41.337351 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:41.337404 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:41.338027 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.338546 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.339543 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.339724 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.339784 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:41.342209 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:41.342210 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:41.342211 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:41.342219 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:41.343637 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:41.344029 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:41.344031 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:41.344031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:41.344089 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:41.344310 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:41.344311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:41.344312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:41.344421 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:41.344498 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:41.344653 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 04:43:41.344757 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 04:43:41.344950 9985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:41.336977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:45703 (local address 127.0.0.1:55546)
0928 04:43:41.337176 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:41.337190 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:41.337278 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:41.337439 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:41.337440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:41.337859 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:41.337920 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:41.337923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.338640 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.338641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:41.339452 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:41.339455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.339809 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.339810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:41.339870 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:41.340578 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:41.340584 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:41.342102 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:41.343741 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:41.343748 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:41.343751 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:41.343952 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:41.344175 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:41.344176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:41.344176 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:41.344236 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:41.344487 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:41.344494 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:41.344541 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 04:43:41.344631 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 04:43:41.344829 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":82,"thread_start_us":95,"threads_started":1}
I20260928 04:43:41.346941 9435 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:55546:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 04:43:41.347174 9435 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:43:41.348829 9572 tablet_service.cc:1467] Processing CreateTablet for tablet cc296916da864a46ae61c1a2917b5b64 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bd7e5468590a418da333f8bd20ffeddd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 04:43:41.348831 9712 tablet_service.cc:1467] Processing CreateTablet for tablet cc296916da864a46ae61c1a2917b5b64 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bd7e5468590a418da333f8bd20ffeddd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 04:43:41.348893 9852 tablet_service.cc:1467] Processing CreateTablet for tablet cc296916da864a46ae61c1a2917b5b64 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bd7e5468590a418da333f8bd20ffeddd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 04:43:41.348984 9572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc296916da864a46ae61c1a2917b5b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:41.348984 9712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc296916da864a46ae61c1a2917b5b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:41.349050 9852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc296916da864a46ae61c1a2917b5b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:41.350507 9946 tablet_bootstrap.cc:492] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809: Bootstrap starting.
I20260928 04:43:41.350653 9948 tablet_bootstrap.cc:492] T cc296916da864a46ae61c1a2917b5b64 P e867826874874526b87f5fef3a61a238: Bootstrap starting.
I20260928 04:43:41.350848 9946 tablet_bootstrap.cc:654] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:41.351091 9948 tablet_bootstrap.cc:654] T cc296916da864a46ae61c1a2917b5b64 P e867826874874526b87f5fef3a61a238: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:41.351717 9948 tablet_bootstrap.cc:492] T cc296916da864a46ae61c1a2917b5b64 P e867826874874526b87f5fef3a61a238: No bootstrap required, opened a new log
I20260928 04:43:41.351770 9948 ts_tablet_manager.cc:1397] T cc296916da864a46ae61c1a2917b5b64 P e867826874874526b87f5fef3a61a238: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 04:43:41.351941 9948 raft_consensus.cc:348] T cc296916da864a46ae61c1a2917b5b64 P e867826874874526b87f5fef3a61a238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.352011 9948 raft_consensus.cc:374] T cc296916da864a46ae61c1a2917b5b64 P e867826874874526b87f5fef3a61a238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:41.352029 9948 raft_consensus.cc:729] T cc296916da864a46ae61c1a2917b5b64 P e867826874874526b87f5fef3a61a238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e867826874874526b87f5fef3a61a238, State: Initialized, Role: FOLLOWER
I20260928 04:43:41.352058 9946 tablet_bootstrap.cc:492] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809: No bootstrap required, opened a new log
I20260928 04:43:41.352069 9948 consensus_queue.cc:260] T cc296916da864a46ae61c1a2917b5b64 P e867826874874526b87f5fef3a61a238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.352102 9946 ts_tablet_manager.cc:1397] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 04:43:41.352145 9948 ts_tablet_manager.cc:1428] T cc296916da864a46ae61c1a2917b5b64 P e867826874874526b87f5fef3a61a238: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 04:43:41.352222 9947 tablet_bootstrap.cc:492] T cc296916da864a46ae61c1a2917b5b64 P fd16e3dcebe3400b8e6985dd10659cf5: Bootstrap starting.
I20260928 04:43:41.352226 9946 raft_consensus.cc:348] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.352286 9946 raft_consensus.cc:374] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:41.352305 9946 raft_consensus.cc:729] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aac7e335b4450da720babd10ed7809, State: Initialized, Role: FOLLOWER
I20260928 04:43:41.352332 9946 consensus_queue.cc:260] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.352406 9946 ts_tablet_manager.cc:1428] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:41.352505 9954 tablet_replica.cc:439] T cc296916da864a46ae61c1a2917b5b64 P e867826874874526b87f5fef3a61a238: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } } }
I20260928 04:43:41.352602 9954 tablet_replica.cc:442] T cc296916da864a46ae61c1a2917b5b64 P e867826874874526b87f5fef3a61a238: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:43:41.352660 9947 tablet_bootstrap.cc:654] T cc296916da864a46ae61c1a2917b5b64 P fd16e3dcebe3400b8e6985dd10659cf5: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:41.352627 9967 tablet_replica.cc:439] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } } }
I20260928 04:43:41.352692 9967 tablet_replica.cc:442] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:43:41.353946 9947 tablet_bootstrap.cc:492] T cc296916da864a46ae61c1a2917b5b64 P fd16e3dcebe3400b8e6985dd10659cf5: No bootstrap required, opened a new log
I20260928 04:43:41.353997 9947 ts_tablet_manager.cc:1397] T cc296916da864a46ae61c1a2917b5b64 P fd16e3dcebe3400b8e6985dd10659cf5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 04:43:41.354144 9947 raft_consensus.cc:348] T cc296916da864a46ae61c1a2917b5b64 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.354228 9947 raft_consensus.cc:374] T cc296916da864a46ae61c1a2917b5b64 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:41.354254 9947 raft_consensus.cc:729] T cc296916da864a46ae61c1a2917b5b64 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd16e3dcebe3400b8e6985dd10659cf5, State: Initialized, Role: FOLLOWER
I20260928 04:43:41.354324 9947 consensus_queue.cc:260] T cc296916da864a46ae61c1a2917b5b64 P fd16e3dcebe3400b8e6985dd10659cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.354419 9947 ts_tablet_manager.cc:1428] T cc296916da864a46ae61c1a2917b5b64 P fd16e3dcebe3400b8e6985dd10659cf5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:41.354473 9978 tablet_replica.cc:439] T cc296916da864a46ae61c1a2917b5b64 P fd16e3dcebe3400b8e6985dd10659cf5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } } }
I20260928 04:43:41.354544 9978 tablet_replica.cc:442] T cc296916da864a46ae61c1a2917b5b64 P fd16e3dcebe3400b8e6985dd10659cf5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:43:41.511752 9967 raft_consensus.cc:482] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:43:41.511860 9967 raft_consensus.cc:504] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.512045 9967 leader_election.cc:288] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd16e3dcebe3400b8e6985dd10659cf5 (127.1.226.193:40603), e867826874874526b87f5fef3a61a238 (127.1.226.195:34427)
I20260928 04:43:41.512305 9592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc296916da864a46ae61c1a2917b5b64" candidate_uuid: "27aac7e335b4450da720babd10ed7809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" is_pre_election: true
I20260928 04:43:41.512328 9872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc296916da864a46ae61c1a2917b5b64" candidate_uuid: "27aac7e335b4450da720babd10ed7809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e867826874874526b87f5fef3a61a238" is_pre_election: true
I20260928 04:43:41.512414 9872 raft_consensus.cc:2454] T cc296916da864a46ae61c1a2917b5b64 P e867826874874526b87f5fef3a61a238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aac7e335b4450da720babd10ed7809 in term 0.
I20260928 04:43:41.512418 9592 raft_consensus.cc:2454] T cc296916da864a46ae61c1a2917b5b64 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aac7e335b4450da720babd10ed7809 in term 0.
I20260928 04:43:41.512604 9665 leader_election.cc:302] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aac7e335b4450da720babd10ed7809, e867826874874526b87f5fef3a61a238; no voters:
I20260928 04:43:41.512701 9967 raft_consensus.cc:2781] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:43:41.512741 9967 raft_consensus.cc:482] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:43:41.512771 9967 raft_consensus.cc:3037] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:41.513506 9967 raft_consensus.cc:504] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.513747 9967 leader_election.cc:288] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809 [CANDIDATE]: Term 1 election: Requested vote from peers fd16e3dcebe3400b8e6985dd10659cf5 (127.1.226.193:40603), e867826874874526b87f5fef3a61a238 (127.1.226.195:34427)
I20260928 04:43:41.513849 9592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc296916da864a46ae61c1a2917b5b64" candidate_uuid: "27aac7e335b4450da720babd10ed7809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd16e3dcebe3400b8e6985dd10659cf5"
I20260928 04:43:41.513943 9592 raft_consensus.cc:3037] T cc296916da864a46ae61c1a2917b5b64 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:41.513911 9872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc296916da864a46ae61c1a2917b5b64" candidate_uuid: "27aac7e335b4450da720babd10ed7809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e867826874874526b87f5fef3a61a238"
I20260928 04:43:41.514017 9872 raft_consensus.cc:3037] T cc296916da864a46ae61c1a2917b5b64 P e867826874874526b87f5fef3a61a238 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:41.514569 9872 raft_consensus.cc:2454] T cc296916da864a46ae61c1a2917b5b64 P e867826874874526b87f5fef3a61a238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aac7e335b4450da720babd10ed7809 in term 1.
I20260928 04:43:41.514619 9592 raft_consensus.cc:2454] T cc296916da864a46ae61c1a2917b5b64 P fd16e3dcebe3400b8e6985dd10659cf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aac7e335b4450da720babd10ed7809 in term 1.
I20260928 04:43:41.514810 9665 leader_election.cc:302] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aac7e335b4450da720babd10ed7809, e867826874874526b87f5fef3a61a238; no voters:
I20260928 04:43:41.514943 9967 raft_consensus.cc:2781] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:41.514992 9967 raft_consensus.cc:686] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809 [term 1 LEADER]: Becoming Leader. State: Replica: 27aac7e335b4450da720babd10ed7809, State: Running, Role: LEADER
I20260928 04:43:41.515048 9967 consensus_queue.cc:237] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.515336 9959 tablet_replica.cc:439] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809: TxnStatusTablet state changed. Reason: New leader 27aac7e335b4450da720babd10ed7809. Latest consensus state: current_term: 1 leader_uuid: "27aac7e335b4450da720babd10ed7809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } } }
I20260928 04:43:41.515430 9959 tablet_replica.cc:442] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809: This TxnStatusTablet replica's current role is: LEADER
I20260928 04:43:41.515617 9990 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 04:43:41.515738 9990 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 04:43:41.515735 9435 catalog_manager.cc:5495] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27aac7e335b4450da720babd10ed7809 (127.1.226.194). New cstate: current_term: 1 leader_uuid: "27aac7e335b4450da720babd10ed7809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } health_report { overall_health: UNKNOWN } } }
I20260928 04:43:41.538443 9944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:41.535590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41226 (local address 127.1.226.194:44519)
0928 04:43:41.535655 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:41.535657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:41.535668 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:43:41.535737 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:41.535738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:41.535746 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:41.535796 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:43:41.535977 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.536510 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.537706 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.537896 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.537965 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:41.538181 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:43:41.538214 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:43:41.538276 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:41.538358 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 04:43:41.538380 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 04:43:41.538492 9985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:41.535515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:44519 (local address 127.0.0.1:41226)
0928 04:43:41.535609 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:41.535620 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:41.535665 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:41.535832 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:41.535833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:41.535835 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:43:41.535898 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:41.535900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.536593 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.536594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:41.537542 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:41.537545 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.537981 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.537982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:41.538053 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:41.538070 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:43:41.538292 (+ 222us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:43:41.538299 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 04:43:41.538354 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 04:43:41.538409 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 04:43:41.539554 9872 raft_consensus.cc:1261] T cc296916da864a46ae61c1a2917b5b64 P e867826874874526b87f5fef3a61a238 [term 1 FOLLOWER]: Refusing update from remote peer 27aac7e335b4450da720babd10ed7809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:41.539724 9592 raft_consensus.cc:1261] T cc296916da864a46ae61c1a2917b5b64 P fd16e3dcebe3400b8e6985dd10659cf5 [term 1 FOLLOWER]: Refusing update from remote peer 27aac7e335b4450da720babd10ed7809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:41.539845 9959 consensus_queue.cc:1035] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:41.539949 9967 consensus_queue.cc:1035] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:41.540460 9954 tablet_replica.cc:439] T cc296916da864a46ae61c1a2917b5b64 P e867826874874526b87f5fef3a61a238: TxnStatusTablet state changed. Reason: New leader 27aac7e335b4450da720babd10ed7809. Latest consensus state: current_term: 1 leader_uuid: "27aac7e335b4450da720babd10ed7809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } } }
I20260928 04:43:41.540555 9954 tablet_replica.cc:442] T cc296916da864a46ae61c1a2917b5b64 P e867826874874526b87f5fef3a61a238: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:43:41.540530 9978 tablet_replica.cc:439] T cc296916da864a46ae61c1a2917b5b64 P fd16e3dcebe3400b8e6985dd10659cf5: TxnStatusTablet state changed. Reason: New leader 27aac7e335b4450da720babd10ed7809. Latest consensus state: current_term: 1 leader_uuid: "27aac7e335b4450da720babd10ed7809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } } }
I20260928 04:43:41.540634 9978 tablet_replica.cc:442] T cc296916da864a46ae61c1a2917b5b64 P fd16e3dcebe3400b8e6985dd10659cf5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:43:41.541165 9962 mvcc.cc:205] Tried to move back new op lower bound from 7334177265823977472 to 7334177265725886464. Current Snapshot: MvccSnapshot[applied={T|T < 7334177265823977472}]
I20260928 04:43:41.541242 9959 tablet_replica.cc:439] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "27aac7e335b4450da720babd10ed7809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } } }
I20260928 04:43:41.541326 9959 tablet_replica.cc:442] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809: This TxnStatusTablet replica's current role is: LEADER
I20260928 04:43:41.541754 9959 tablet_replica.cc:439] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "27aac7e335b4450da720babd10ed7809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } } }
I20260928 04:43:41.541826 9959 tablet_replica.cc:442] T cc296916da864a46ae61c1a2917b5b64 P 27aac7e335b4450da720babd10ed7809: This TxnStatusTablet replica's current role is: LEADER
I20260928 04:43:41.542371 9978 tablet_replica.cc:439] T cc296916da864a46ae61c1a2917b5b64 P fd16e3dcebe3400b8e6985dd10659cf5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "27aac7e335b4450da720babd10ed7809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } } }
I20260928 04:43:41.542454 9978 tablet_replica.cc:442] T cc296916da864a46ae61c1a2917b5b64 P fd16e3dcebe3400b8e6985dd10659cf5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:43:41.542755 9954 tablet_replica.cc:439] T cc296916da864a46ae61c1a2917b5b64 P e867826874874526b87f5fef3a61a238: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "27aac7e335b4450da720babd10ed7809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } } }
I20260928 04:43:41.542850 9954 tablet_replica.cc:442] T cc296916da864a46ae61c1a2917b5b64 P e867826874874526b87f5fef3a61a238: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:43:41.547396 9945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:41.544496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38988 (local address 127.1.226.193:40603)
0928 04:43:41.544545 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:41.544547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:41.544558 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:43:41.544671 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:41.544672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:41.544698 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:41.544797 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:43:41.544939 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.545640 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.546421 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.546725 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.546806 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:41.547023 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:43:41.547160 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:43:41.547246 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:41.547291 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 04:43:41.547318 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 04:43:41.553236 9944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:41.550273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:47433 (local address 127.1.226.194:44519)
0928 04:43:41.550313 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:41.550315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:41.550435 (+ 120us) server_negotiation.cc:407] Connection header received
0928 04:43:41.550506 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:41.550506 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:41.550515 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:41.550562 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:43:41.550719 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.551193 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.552340 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.552540 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.552603 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:41.552879 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:43:41.552911 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:43:41.552959 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:41.553143 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 04:43:41.553167 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 04:43:41.553643 9997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:41.550206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:44519 (local address 127.1.226.193:47433)
0928 04:43:41.550368 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:41.550379 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:41.550440 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:41.550568 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:41.550569 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:41.550570 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:43:41.550647 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:41.550648 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.551280 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.551281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:41.552249 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:41.552252 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.552677 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.552678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:41.552754 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:41.552779 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:43:41.553023 (+ 244us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:43:41.553026 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:43:41.553097 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 04:43:41.553173 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260928 04:43:41.556547 9872 raft_consensus.cc:1261] T 6a768c67cc5f436fa2efa34bd4217c26 P e867826874874526b87f5fef3a61a238 [term 1 FOLLOWER]: Refusing update from remote peer fd16e3dcebe3400b8e6985dd10659cf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:41.556577 9732 raft_consensus.cc:1261] T 6a768c67cc5f436fa2efa34bd4217c26 P 27aac7e335b4450da720babd10ed7809 [term 1 FOLLOWER]: Refusing update from remote peer fd16e3dcebe3400b8e6985dd10659cf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:41.556783 9978 consensus_queue.cc:1035] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:41.556938 9953 consensus_queue.cc:1035] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:41.569242 9945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:41.566319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:47259 (local address 127.1.226.193:40603)
0928 04:43:41.566399 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:41.566402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:41.566552 (+ 150us) server_negotiation.cc:407] Connection header received
0928 04:43:41.566577 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:41.566578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:41.566587 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:41.566656 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:43:41.566782 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.567262 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.568317 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.568659 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.568739 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:41.568974 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:43:41.569015 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:43:41.569075 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:41.569144 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 04:43:41.569166 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 04:43:41.569303 10001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:41.566233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:40603 (local address 127.1.226.194:47259)
0928 04:43:41.566407 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:41.566417 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:41.566493 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:41.566670 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:41.566671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:41.566673 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:43:41.566728 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:41.566730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.567345 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.567346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:41.568012 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:41.568015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.568751 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.568752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:41.568833 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:41.568852 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:43:41.569090 (+ 238us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:43:41.569092 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:43:41.569130 (+ 38us) client_negotiation.cc:240] Negotiation successful
0928 04:43:41.569179 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
I20260928 04:43:41.578842 9435 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:43:41.578971 9435 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:43:41.579541 9435 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c4ba4735141419a8026489a0785daa9: Sending DeleteTablet for 3 replicas of tablet 6a768c67cc5f436fa2efa34bd4217c26
I20260928 04:43:41.579819 9712 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a768c67cc5f436fa2efa34bd4217c26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:41 UTC) from {username='oryx', principal='oryx/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:41202
I20260928 04:43:41.579840 9852 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a768c67cc5f436fa2efa34bd4217c26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:41 UTC) from {username='oryx', principal='oryx/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:50672
I20260928 04:43:41.579842 9572 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a768c67cc5f436fa2efa34bd4217c26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:41 UTC) from {username='oryx', principal='oryx/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:38978
I20260928 04:43:41.579933 9972 tablet_replica.cc:330] T 6a768c67cc5f436fa2efa34bd4217c26 P 27aac7e335b4450da720babd10ed7809: stopping tablet replica
I20260928 04:43:41.579936 9973 tablet_replica.cc:330] T 6a768c67cc5f436fa2efa34bd4217c26 P e867826874874526b87f5fef3a61a238: stopping tablet replica
I20260928 04:43:41.579990 9972 raft_consensus.cc:2229] T 6a768c67cc5f436fa2efa34bd4217c26 P 27aac7e335b4450da720babd10ed7809 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:43:41.579994 9973 raft_consensus.cc:2229] T 6a768c67cc5f436fa2efa34bd4217c26 P e867826874874526b87f5fef3a61a238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:43:41.580032 9973 raft_consensus.cc:2258] T 6a768c67cc5f436fa2efa34bd4217c26 P e867826874874526b87f5fef3a61a238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:41.580036 9972 raft_consensus.cc:2258] T 6a768c67cc5f436fa2efa34bd4217c26 P 27aac7e335b4450da720babd10ed7809 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:41.580111 9974 tablet_replica.cc:330] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5: stopping tablet replica
I20260928 04:43:41.580197 9974 raft_consensus.cc:2229] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:43:41.580271 9974 raft_consensus.cc:2258] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:41.580701 9974 ts_tablet_manager.cc:1905] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:41.580793 9973 ts_tablet_manager.cc:1905] T 6a768c67cc5f436fa2efa34bd4217c26 P e867826874874526b87f5fef3a61a238: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:41.582490 9973 ts_tablet_manager.cc:1918] T 6a768c67cc5f436fa2efa34bd4217c26 P e867826874874526b87f5fef3a61a238: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:43:41.582545 9973 log.cc:1196] T 6a768c67cc5f436fa2efa34bd4217c26 P e867826874874526b87f5fef3a61a238: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-2/wal/wals/6a768c67cc5f436fa2efa34bd4217c26
I20260928 04:43:41.582602 9974 ts_tablet_manager.cc:1918] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:43:41.582657 9974 log.cc:1196] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-0/wal/wals/6a768c67cc5f436fa2efa34bd4217c26
I20260928 04:43:41.582795 9973 ts_tablet_manager.cc:1939] T 6a768c67cc5f436fa2efa34bd4217c26 P e867826874874526b87f5fef3a61a238: Deleting consensus metadata
I20260928 04:43:41.582885 9974 ts_tablet_manager.cc:1939] T 6a768c67cc5f436fa2efa34bd4217c26 P fd16e3dcebe3400b8e6985dd10659cf5: Deleting consensus metadata
I20260928 04:43:41.583189 9419 catalog_manager.cc:4841] TS fd16e3dcebe3400b8e6985dd10659cf5 (127.1.226.193:40603): tablet 6a768c67cc5f436fa2efa34bd4217c26 (table test-table [id=8c2b96ba30ce41fdba7e24c34ad8915d]) successfully deleted
I20260928 04:43:41.583194 9972 ts_tablet_manager.cc:1905] T 6a768c67cc5f436fa2efa34bd4217c26 P 27aac7e335b4450da720babd10ed7809: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:41.584779 9972 ts_tablet_manager.cc:1918] T 6a768c67cc5f436fa2efa34bd4217c26 P 27aac7e335b4450da720babd10ed7809: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:43:41.584847 9972 log.cc:1196] T 6a768c67cc5f436fa2efa34bd4217c26 P 27aac7e335b4450da720babd10ed7809: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-1/wal/wals/6a768c67cc5f436fa2efa34bd4217c26
I20260928 04:43:41.585079 9972 ts_tablet_manager.cc:1939] T 6a768c67cc5f436fa2efa34bd4217c26 P 27aac7e335b4450da720babd10ed7809: Deleting consensus metadata
I20260928 04:43:41.585436 9422 catalog_manager.cc:4841] TS 27aac7e335b4450da720babd10ed7809 (127.1.226.194:44519): tablet 6a768c67cc5f436fa2efa34bd4217c26 (table test-table [id=8c2b96ba30ce41fdba7e24c34ad8915d]) successfully deleted
I20260928 04:43:41.585554 9421 catalog_manager.cc:4841] TS e867826874874526b87f5fef3a61a238 (127.1.226.195:34427): tablet 6a768c67cc5f436fa2efa34bd4217c26 (table test-table [id=8c2b96ba30ce41fdba7e24c34ad8915d]) successfully deleted
Sep 28 04:43:41 dist-test-slave-j5c3 krb5kdc[9390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 04:43:41 dist-test-slave-j5c3 krb5kdc[9390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:41 dist-test-slave-j5c3 krb5kdc[9390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 04:43:41.604712 9484 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:41.599161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55560 (local address 127.1.226.254:45703)
0928 04:43:41.599223 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:41.599225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:41.599359 (+ 134us) server_negotiation.cc:407] Connection header received
0928 04:43:41.599442 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:41.599443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:41.599454 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:41.599528 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:41.600111 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.600678 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.601455 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.601658 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.601753 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:41.604642 (+ 2889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55560: BlockingRecv error: recv got EOF from 127.0.0.1:55560 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 04:43:41 dist-test-slave-j5c3 krb5kdc[9390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM
I20260928 04:43:41.614159 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:41.605818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55562 (local address 127.1.226.254:45703)
0928 04:43:41.605867 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:41.605869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:41.606116 (+ 247us) server_negotiation.cc:407] Connection header received
0928 04:43:41.606177 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:41.606178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:41.606188 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:41.606269 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:41.606736 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.607208 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.607913 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.608104 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.608230 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:41.610369 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:41.610370 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:41.610371 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:41.610378 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:41.611787 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:41.612186 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:41.612188 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:41.612189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:41.612249 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:41.612457 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:41.612458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:41.612458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:41.612572 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:41.612678 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:41.612792 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 04:43:41.612872 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:43:41.615806 9435 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:43:41.615993 9435 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:43:41.617316 9572 tablet_service.cc:1467] Processing CreateTablet for tablet a12b062c315e4969b01eb630a41af4b5 (DEFAULT_TABLE table=test-table [id=80ebeff572f6402891dc5767aa60d842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:41.617358 9852 tablet_service.cc:1467] Processing CreateTablet for tablet a12b062c315e4969b01eb630a41af4b5 (DEFAULT_TABLE table=test-table [id=80ebeff572f6402891dc5767aa60d842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:41.617458 9572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a12b062c315e4969b01eb630a41af4b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:41.617473 9852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a12b062c315e4969b01eb630a41af4b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:41.618767 9948 tablet_bootstrap.cc:492] T a12b062c315e4969b01eb630a41af4b5 P e867826874874526b87f5fef3a61a238: Bootstrap starting.
I20260928 04:43:41.618777 9947 tablet_bootstrap.cc:492] T a12b062c315e4969b01eb630a41af4b5 P fd16e3dcebe3400b8e6985dd10659cf5: Bootstrap starting.
I20260928 04:43:41.619165 9948 tablet_bootstrap.cc:654] T a12b062c315e4969b01eb630a41af4b5 P e867826874874526b87f5fef3a61a238: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:41.619283 9947 tablet_bootstrap.cc:654] T a12b062c315e4969b01eb630a41af4b5 P fd16e3dcebe3400b8e6985dd10659cf5: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:41.619848 9712 tablet_service.cc:1467] Processing CreateTablet for tablet a12b062c315e4969b01eb630a41af4b5 (DEFAULT_TABLE table=test-table [id=80ebeff572f6402891dc5767aa60d842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:41.619933 9948 tablet_bootstrap.cc:492] T a12b062c315e4969b01eb630a41af4b5 P e867826874874526b87f5fef3a61a238: No bootstrap required, opened a new log
I20260928 04:43:41.619966 9712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a12b062c315e4969b01eb630a41af4b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:41.619972 9948 ts_tablet_manager.cc:1397] T a12b062c315e4969b01eb630a41af4b5 P e867826874874526b87f5fef3a61a238: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:41.620107 9948 raft_consensus.cc:348] T a12b062c315e4969b01eb630a41af4b5 P e867826874874526b87f5fef3a61a238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.620240 9948 raft_consensus.cc:374] T a12b062c315e4969b01eb630a41af4b5 P e867826874874526b87f5fef3a61a238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:41.620263 9948 raft_consensus.cc:729] T a12b062c315e4969b01eb630a41af4b5 P e867826874874526b87f5fef3a61a238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e867826874874526b87f5fef3a61a238, State: Initialized, Role: FOLLOWER
I20260928 04:43:41.620302 9948 consensus_queue.cc:260] T a12b062c315e4969b01eb630a41af4b5 P e867826874874526b87f5fef3a61a238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.620388 9948 ts_tablet_manager.cc:1428] T a12b062c315e4969b01eb630a41af4b5 P e867826874874526b87f5fef3a61a238: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 04:43:41.621295 9946 tablet_bootstrap.cc:492] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809: Bootstrap starting.
I20260928 04:43:41.621673 9946 tablet_bootstrap.cc:654] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:41.622337 9946 tablet_bootstrap.cc:492] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809: No bootstrap required, opened a new log
I20260928 04:43:41.622408 9946 ts_tablet_manager.cc:1397] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:41.622560 9946 raft_consensus.cc:348] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.622634 9946 raft_consensus.cc:374] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:41.622663 9946 raft_consensus.cc:729] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aac7e335b4450da720babd10ed7809, State: Initialized, Role: FOLLOWER
I20260928 04:43:41.622704 9946 consensus_queue.cc:260] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.622798 9946 ts_tablet_manager.cc:1428] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:41.624572 9947 tablet_bootstrap.cc:492] T a12b062c315e4969b01eb630a41af4b5 P fd16e3dcebe3400b8e6985dd10659cf5: No bootstrap required, opened a new log
I20260928 04:43:41.624640 9947 ts_tablet_manager.cc:1397] T a12b062c315e4969b01eb630a41af4b5 P fd16e3dcebe3400b8e6985dd10659cf5: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260928 04:43:41.624799 9947 raft_consensus.cc:348] T a12b062c315e4969b01eb630a41af4b5 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.624873 9947 raft_consensus.cc:374] T a12b062c315e4969b01eb630a41af4b5 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:41.624900 9947 raft_consensus.cc:729] T a12b062c315e4969b01eb630a41af4b5 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd16e3dcebe3400b8e6985dd10659cf5, State: Initialized, Role: FOLLOWER
I20260928 04:43:41.624960 9947 consensus_queue.cc:260] T a12b062c315e4969b01eb630a41af4b5 P fd16e3dcebe3400b8e6985dd10659cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.625051 9947 ts_tablet_manager.cc:1428] T a12b062c315e4969b01eb630a41af4b5 P fd16e3dcebe3400b8e6985dd10659cf5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 04:43:41.915817 9967 raft_consensus.cc:482] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:43:41.915941 9967 raft_consensus.cc:504] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.916201 9967 leader_election.cc:288] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd16e3dcebe3400b8e6985dd10659cf5 (127.1.226.193:40603), e867826874874526b87f5fef3a61a238 (127.1.226.195:34427)
I20260928 04:43:41.916428 9592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b062c315e4969b01eb630a41af4b5" candidate_uuid: "27aac7e335b4450da720babd10ed7809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" is_pre_election: true
I20260928 04:43:41.916464 9872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b062c315e4969b01eb630a41af4b5" candidate_uuid: "27aac7e335b4450da720babd10ed7809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e867826874874526b87f5fef3a61a238" is_pre_election: true
I20260928 04:43:41.916556 9592 raft_consensus.cc:2454] T a12b062c315e4969b01eb630a41af4b5 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aac7e335b4450da720babd10ed7809 in term 0.
I20260928 04:43:41.916553 9872 raft_consensus.cc:2454] T a12b062c315e4969b01eb630a41af4b5 P e867826874874526b87f5fef3a61a238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27aac7e335b4450da720babd10ed7809 in term 0.
I20260928 04:43:41.916752 9663 leader_election.cc:302] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aac7e335b4450da720babd10ed7809, fd16e3dcebe3400b8e6985dd10659cf5; no voters:
I20260928 04:43:41.916844 9967 raft_consensus.cc:2781] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:43:41.916887 9967 raft_consensus.cc:482] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:43:41.916906 9967 raft_consensus.cc:3037] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:41.917900 9967 raft_consensus.cc:504] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.918059 9967 leader_election.cc:288] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809 [CANDIDATE]: Term 1 election: Requested vote from peers fd16e3dcebe3400b8e6985dd10659cf5 (127.1.226.193:40603), e867826874874526b87f5fef3a61a238 (127.1.226.195:34427)
I20260928 04:43:41.918226 9592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b062c315e4969b01eb630a41af4b5" candidate_uuid: "27aac7e335b4450da720babd10ed7809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd16e3dcebe3400b8e6985dd10659cf5"
I20260928 04:43:41.918314 9592 raft_consensus.cc:3037] T a12b062c315e4969b01eb630a41af4b5 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:41.918301 9872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12b062c315e4969b01eb630a41af4b5" candidate_uuid: "27aac7e335b4450da720babd10ed7809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e867826874874526b87f5fef3a61a238"
I20260928 04:43:41.918377 9872 raft_consensus.cc:3037] T a12b062c315e4969b01eb630a41af4b5 P e867826874874526b87f5fef3a61a238 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:41.919188 9592 raft_consensus.cc:2454] T a12b062c315e4969b01eb630a41af4b5 P fd16e3dcebe3400b8e6985dd10659cf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aac7e335b4450da720babd10ed7809 in term 1.
I20260928 04:43:41.919201 9872 raft_consensus.cc:2454] T a12b062c315e4969b01eb630a41af4b5 P e867826874874526b87f5fef3a61a238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27aac7e335b4450da720babd10ed7809 in term 1.
I20260928 04:43:41.919384 9663 leader_election.cc:302] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aac7e335b4450da720babd10ed7809, fd16e3dcebe3400b8e6985dd10659cf5; no voters:
I20260928 04:43:41.919497 9967 raft_consensus.cc:2781] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:41.919591 9967 raft_consensus.cc:686] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809 [term 1 LEADER]: Becoming Leader. State: Replica: 27aac7e335b4450da720babd10ed7809, State: Running, Role: LEADER
I20260928 04:43:41.919652 9967 consensus_queue.cc:237] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.920271 9435 catalog_manager.cc:5495] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27aac7e335b4450da720babd10ed7809 (127.1.226.194). New cstate: current_term: 1 leader_uuid: "27aac7e335b4450da720babd10ed7809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } health_report { overall_health: UNKNOWN } } }
I20260928 04:43:41.978996 9944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:41.976513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41228 (local address 127.1.226.194:44519)
0928 04:43:41.976585 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:41.976587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:41.976597 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:43:41.976648 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:41.976649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:41.976661 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:41.976729 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:43:41.976866 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.977412 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.978261 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:41.978467 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:41.978548 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:41.978707 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:43:41.978737 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:43:41.978783 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:41.978852 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 04:43:41.978870 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 04:43:41.979945 9872 raft_consensus.cc:1261] T a12b062c315e4969b01eb630a41af4b5 P e867826874874526b87f5fef3a61a238 [term 1 FOLLOWER]: Refusing update from remote peer 27aac7e335b4450da720babd10ed7809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:41.979976 9592 raft_consensus.cc:1261] T a12b062c315e4969b01eb630a41af4b5 P fd16e3dcebe3400b8e6985dd10659cf5 [term 1 FOLLOWER]: Refusing update from remote peer 27aac7e335b4450da720babd10ed7809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:41.980342 9959 consensus_queue.cc:1035] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:41.980456 9967 consensus_queue.cc:1035] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:41.983654 9435 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:43:41.983757 9435 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:43:41.984483 9435 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c4ba4735141419a8026489a0785daa9: Sending DeleteTablet for 3 replicas of tablet a12b062c315e4969b01eb630a41af4b5
I20260928 04:43:41.984720 9712 tablet_service.cc:1514] Processing DeleteTablet for tablet a12b062c315e4969b01eb630a41af4b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:41 UTC) from {username='oryx', principal='oryx/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:41202
I20260928 04:43:41.984771 9852 tablet_service.cc:1514] Processing DeleteTablet for tablet a12b062c315e4969b01eb630a41af4b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:41 UTC) from {username='oryx', principal='oryx/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:50672
I20260928 04:43:41.984774 9572 tablet_service.cc:1514] Processing DeleteTablet for tablet a12b062c315e4969b01eb630a41af4b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:41 UTC) from {username='oryx', principal='oryx/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:38978
I20260928 04:43:41.984813 9972 tablet_replica.cc:330] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809: stopping tablet replica
I20260928 04:43:41.984870 9972 raft_consensus.cc:2229] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:43:41.984876 9974 tablet_replica.cc:330] T a12b062c315e4969b01eb630a41af4b5 P fd16e3dcebe3400b8e6985dd10659cf5: stopping tablet replica
I20260928 04:43:41.984925 9972 raft_consensus.cc:2258] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:41.984925 9974 raft_consensus.cc:2229] T a12b062c315e4969b01eb630a41af4b5 P fd16e3dcebe3400b8e6985dd10659cf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:43:41.984993 9974 raft_consensus.cc:2258] T a12b062c315e4969b01eb630a41af4b5 P fd16e3dcebe3400b8e6985dd10659cf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:41.985078 9973 tablet_replica.cc:330] T a12b062c315e4969b01eb630a41af4b5 P e867826874874526b87f5fef3a61a238: stopping tablet replica
I20260928 04:43:41.985127 9973 raft_consensus.cc:2229] T a12b062c315e4969b01eb630a41af4b5 P e867826874874526b87f5fef3a61a238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:43:41.985170 9973 raft_consensus.cc:2258] T a12b062c315e4969b01eb630a41af4b5 P e867826874874526b87f5fef3a61a238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:41.985203 9972 ts_tablet_manager.cc:1905] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:41.985571 9973 ts_tablet_manager.cc:1905] T a12b062c315e4969b01eb630a41af4b5 P e867826874874526b87f5fef3a61a238: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:41.985526 9435 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:43:41.985675 9435 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:43:41.986910 9712 tablet_service.cc:1467] Processing CreateTablet for tablet 519c7cb40eaa4e6495bacd020b55f040 (DEFAULT_TABLE table=test-table [id=cf73d5bb4c6641a2b0fb1ebf0539c0c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:41.986987 9572 tablet_service.cc:1467] Processing CreateTablet for tablet 519c7cb40eaa4e6495bacd020b55f040 (DEFAULT_TABLE table=test-table [id=cf73d5bb4c6641a2b0fb1ebf0539c0c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:41.987051 9972 ts_tablet_manager.cc:1918] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:43:41.987082 9712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 519c7cb40eaa4e6495bacd020b55f040. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:41.987097 9972 log.cc:1196] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-1/wal/wals/a12b062c315e4969b01eb630a41af4b5
I20260928 04:43:41.987113 9572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 519c7cb40eaa4e6495bacd020b55f040. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:41.987301 9973 ts_tablet_manager.cc:1918] T a12b062c315e4969b01eb630a41af4b5 P e867826874874526b87f5fef3a61a238: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:43:41.987319 9972 ts_tablet_manager.cc:1939] T a12b062c315e4969b01eb630a41af4b5 P 27aac7e335b4450da720babd10ed7809: Deleting consensus metadata
I20260928 04:43:41.987349 9973 log.cc:1196] T a12b062c315e4969b01eb630a41af4b5 P e867826874874526b87f5fef3a61a238: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-2/wal/wals/a12b062c315e4969b01eb630a41af4b5
I20260928 04:43:41.987571 9973 ts_tablet_manager.cc:1939] T a12b062c315e4969b01eb630a41af4b5 P e867826874874526b87f5fef3a61a238: Deleting consensus metadata
I20260928 04:43:41.987612 9422 catalog_manager.cc:4841] TS 27aac7e335b4450da720babd10ed7809 (127.1.226.194:44519): tablet a12b062c315e4969b01eb630a41af4b5 (table test-table [id=80ebeff572f6402891dc5767aa60d842]) successfully deleted
I20260928 04:43:41.988387 9974 ts_tablet_manager.cc:1905] T a12b062c315e4969b01eb630a41af4b5 P fd16e3dcebe3400b8e6985dd10659cf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:41.988588 9946 tablet_bootstrap.cc:492] T 519c7cb40eaa4e6495bacd020b55f040 P 27aac7e335b4450da720babd10ed7809: Bootstrap starting.
I20260928 04:43:41.988930 9946 tablet_bootstrap.cc:654] T 519c7cb40eaa4e6495bacd020b55f040 P 27aac7e335b4450da720babd10ed7809: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:41.989202 9947 tablet_bootstrap.cc:492] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5: Bootstrap starting.
I20260928 04:43:41.989490 9946 tablet_bootstrap.cc:492] T 519c7cb40eaa4e6495bacd020b55f040 P 27aac7e335b4450da720babd10ed7809: No bootstrap required, opened a new log
I20260928 04:43:41.989522 9946 ts_tablet_manager.cc:1397] T 519c7cb40eaa4e6495bacd020b55f040 P 27aac7e335b4450da720babd10ed7809: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:41.989516 9947 tablet_bootstrap.cc:654] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:41.989605 9946 raft_consensus.cc:348] T 519c7cb40eaa4e6495bacd020b55f040 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.989653 9946 raft_consensus.cc:374] T 519c7cb40eaa4e6495bacd020b55f040 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:41.989666 9946 raft_consensus.cc:729] T 519c7cb40eaa4e6495bacd020b55f040 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27aac7e335b4450da720babd10ed7809, State: Initialized, Role: FOLLOWER
I20260928 04:43:41.989729 9946 consensus_queue.cc:260] T 519c7cb40eaa4e6495bacd020b55f040 P 27aac7e335b4450da720babd10ed7809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.989843 9946 ts_tablet_manager.cc:1428] T 519c7cb40eaa4e6495bacd020b55f040 P 27aac7e335b4450da720babd10ed7809: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 04:43:41.989948 9974 ts_tablet_manager.cc:1918] T a12b062c315e4969b01eb630a41af4b5 P fd16e3dcebe3400b8e6985dd10659cf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:43:41.990005 9974 log.cc:1196] T a12b062c315e4969b01eb630a41af4b5 P fd16e3dcebe3400b8e6985dd10659cf5: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-0/wal/wals/a12b062c315e4969b01eb630a41af4b5
I20260928 04:43:41.990156 9947 tablet_bootstrap.cc:492] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5: No bootstrap required, opened a new log
I20260928 04:43:41.990199 9947 ts_tablet_manager.cc:1397] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 04:43:41.990253 9974 ts_tablet_manager.cc:1939] T a12b062c315e4969b01eb630a41af4b5 P fd16e3dcebe3400b8e6985dd10659cf5: Deleting consensus metadata
I20260928 04:43:41.990326 9947 raft_consensus.cc:348] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.990396 9947 raft_consensus.cc:374] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:41.990417 9947 raft_consensus.cc:729] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd16e3dcebe3400b8e6985dd10659cf5, State: Initialized, Role: FOLLOWER
I20260928 04:43:41.990449 9947 consensus_queue.cc:260] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.990549 9947 ts_tablet_manager.cc:1428] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:41.990599 9419 catalog_manager.cc:4841] TS fd16e3dcebe3400b8e6985dd10659cf5 (127.1.226.193:40603): tablet a12b062c315e4969b01eb630a41af4b5 (table test-table [id=80ebeff572f6402891dc5767aa60d842]) successfully deleted
I20260928 04:43:41.990659 9421 catalog_manager.cc:4841] TS e867826874874526b87f5fef3a61a238 (127.1.226.195:34427): tablet a12b062c315e4969b01eb630a41af4b5 (table test-table [id=80ebeff572f6402891dc5767aa60d842]) successfully deleted
I20260928 04:43:41.990763 9852 tablet_service.cc:1467] Processing CreateTablet for tablet 519c7cb40eaa4e6495bacd020b55f040 (DEFAULT_TABLE table=test-table [id=cf73d5bb4c6641a2b0fb1ebf0539c0c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:43:41.990840 9852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 519c7cb40eaa4e6495bacd020b55f040. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:41.991744 9948 tablet_bootstrap.cc:492] T 519c7cb40eaa4e6495bacd020b55f040 P e867826874874526b87f5fef3a61a238: Bootstrap starting.
I20260928 04:43:41.992079 9948 tablet_bootstrap.cc:654] T 519c7cb40eaa4e6495bacd020b55f040 P e867826874874526b87f5fef3a61a238: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:41.993134 9948 tablet_bootstrap.cc:492] T 519c7cb40eaa4e6495bacd020b55f040 P e867826874874526b87f5fef3a61a238: No bootstrap required, opened a new log
I20260928 04:43:41.993203 9948 ts_tablet_manager.cc:1397] T 519c7cb40eaa4e6495bacd020b55f040 P e867826874874526b87f5fef3a61a238: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:41.993355 9948 raft_consensus.cc:348] T 519c7cb40eaa4e6495bacd020b55f040 P e867826874874526b87f5fef3a61a238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.993431 9948 raft_consensus.cc:374] T 519c7cb40eaa4e6495bacd020b55f040 P e867826874874526b87f5fef3a61a238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:41.993474 9948 raft_consensus.cc:729] T 519c7cb40eaa4e6495bacd020b55f040 P e867826874874526b87f5fef3a61a238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e867826874874526b87f5fef3a61a238, State: Initialized, Role: FOLLOWER
I20260928 04:43:41.993541 9948 consensus_queue.cc:260] T 519c7cb40eaa4e6495bacd020b55f040 P e867826874874526b87f5fef3a61a238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:41.993635 9948 ts_tablet_manager.cc:1428] T 519c7cb40eaa4e6495bacd020b55f040 P e867826874874526b87f5fef3a61a238: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:42.229394 9978 raft_consensus.cc:482] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:43:42.229494 9978 raft_consensus.cc:504] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:42.229806 9978 leader_election.cc:288] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27aac7e335b4450da720babd10ed7809 (127.1.226.194:44519), e867826874874526b87f5fef3a61a238 (127.1.226.195:34427)
I20260928 04:43:42.230072 9872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "519c7cb40eaa4e6495bacd020b55f040" candidate_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e867826874874526b87f5fef3a61a238" is_pre_election: true
I20260928 04:43:42.230060 9732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "519c7cb40eaa4e6495bacd020b55f040" candidate_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aac7e335b4450da720babd10ed7809" is_pre_election: true
I20260928 04:43:42.230181 9732 raft_consensus.cc:2454] T 519c7cb40eaa4e6495bacd020b55f040 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd16e3dcebe3400b8e6985dd10659cf5 in term 0.
I20260928 04:43:42.230185 9872 raft_consensus.cc:2454] T 519c7cb40eaa4e6495bacd020b55f040 P e867826874874526b87f5fef3a61a238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd16e3dcebe3400b8e6985dd10659cf5 in term 0.
I20260928 04:43:42.230367 9525 leader_election.cc:302] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e867826874874526b87f5fef3a61a238, fd16e3dcebe3400b8e6985dd10659cf5; no voters:
I20260928 04:43:42.230495 9978 raft_consensus.cc:2781] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:43:42.230554 9978 raft_consensus.cc:482] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:43:42.230578 9978 raft_consensus.cc:3037] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:42.231357 9978 raft_consensus.cc:504] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:42.231499 9978 leader_election.cc:288] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5 [CANDIDATE]: Term 1 election: Requested vote from peers 27aac7e335b4450da720babd10ed7809 (127.1.226.194:44519), e867826874874526b87f5fef3a61a238 (127.1.226.195:34427)
I20260928 04:43:42.231660 9732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "519c7cb40eaa4e6495bacd020b55f040" candidate_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27aac7e335b4450da720babd10ed7809"
I20260928 04:43:42.231752 9732 raft_consensus.cc:3037] T 519c7cb40eaa4e6495bacd020b55f040 P 27aac7e335b4450da720babd10ed7809 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:42.231761 9872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "519c7cb40eaa4e6495bacd020b55f040" candidate_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e867826874874526b87f5fef3a61a238"
I20260928 04:43:42.231842 9872 raft_consensus.cc:3037] T 519c7cb40eaa4e6495bacd020b55f040 P e867826874874526b87f5fef3a61a238 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:42.232486 9732 raft_consensus.cc:2454] T 519c7cb40eaa4e6495bacd020b55f040 P 27aac7e335b4450da720babd10ed7809 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd16e3dcebe3400b8e6985dd10659cf5 in term 1.
I20260928 04:43:42.232545 9872 raft_consensus.cc:2454] T 519c7cb40eaa4e6495bacd020b55f040 P e867826874874526b87f5fef3a61a238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd16e3dcebe3400b8e6985dd10659cf5 in term 1.
I20260928 04:43:42.232681 9526 leader_election.cc:302] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27aac7e335b4450da720babd10ed7809, fd16e3dcebe3400b8e6985dd10659cf5; no voters:
I20260928 04:43:42.232836 9978 raft_consensus.cc:2781] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:42.232962 9978 raft_consensus.cc:686] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5 [term 1 LEADER]: Becoming Leader. State: Replica: fd16e3dcebe3400b8e6985dd10659cf5, State: Running, Role: LEADER
I20260928 04:43:42.233021 9978 consensus_queue.cc:237] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } }
I20260928 04:43:42.233592 9434 catalog_manager.cc:5495] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd16e3dcebe3400b8e6985dd10659cf5 (127.1.226.193). New cstate: current_term: 1 leader_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd16e3dcebe3400b8e6985dd10659cf5" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 40603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 } health_report { overall_health: UNKNOWN } } }
I20260928 04:43:42.280280 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:42.277934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39002 (local address 127.1.226.193:40603)
0928 04:43:42.278065 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:42.278067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:42.278073 (+ 6us) server_negotiation.cc:407] Connection header received
0928 04:43:42.278091 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:42.278092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:42.278102 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:42.278173 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:43:42.278291 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:42.278795 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:42.279531 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:42.279737 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:42.279827 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:42.279983 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:43:42.280021 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:43:42.280081 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:42.280141 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 04:43:42.280185 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260928 04:43:42.282548 9872 raft_consensus.cc:1261] T 519c7cb40eaa4e6495bacd020b55f040 P e867826874874526b87f5fef3a61a238 [term 1 FOLLOWER]: Refusing update from remote peer fd16e3dcebe3400b8e6985dd10659cf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:42.282631 9732 raft_consensus.cc:1261] T 519c7cb40eaa4e6495bacd020b55f040 P 27aac7e335b4450da720babd10ed7809 [term 1 FOLLOWER]: Refusing update from remote peer fd16e3dcebe3400b8e6985dd10659cf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:43:42.282833 10027 consensus_queue.cc:1035] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e867826874874526b87f5fef3a61a238" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:42.282953 9978 consensus_queue.cc:1035] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27aac7e335b4450da720babd10ed7809" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:43:42.293219 9434 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:43:42.293326 9434 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:43:42.293939 9434 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c4ba4735141419a8026489a0785daa9: Sending DeleteTablet for 3 replicas of tablet 519c7cb40eaa4e6495bacd020b55f040
I20260928 04:43:42.294204 9712 tablet_service.cc:1514] Processing DeleteTablet for tablet 519c7cb40eaa4e6495bacd020b55f040 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:42 UTC) from {username='oryx', principal='oryx/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:41202
I20260928 04:43:42.294234 9572 tablet_service.cc:1514] Processing DeleteTablet for tablet 519c7cb40eaa4e6495bacd020b55f040 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:42 UTC) from {username='oryx', principal='oryx/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:38978
I20260928 04:43:42.294304 9972 tablet_replica.cc:330] T 519c7cb40eaa4e6495bacd020b55f040 P 27aac7e335b4450da720babd10ed7809: stopping tablet replica
I20260928 04:43:42.294312 9974 tablet_replica.cc:330] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5: stopping tablet replica
I20260928 04:43:42.294292 9852 tablet_service.cc:1514] Processing DeleteTablet for tablet 519c7cb40eaa4e6495bacd020b55f040 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:43:42 UTC) from {username='oryx', principal='oryx/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:50672
I20260928 04:43:42.294355 9974 raft_consensus.cc:2229] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:43:42.294355 9972 raft_consensus.cc:2229] T 519c7cb40eaa4e6495bacd020b55f040 P 27aac7e335b4450da720babd10ed7809 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:43:42.294387 9973 tablet_replica.cc:330] T 519c7cb40eaa4e6495bacd020b55f040 P e867826874874526b87f5fef3a61a238: stopping tablet replica
I20260928 04:43:42.294405 9972 raft_consensus.cc:2258] T 519c7cb40eaa4e6495bacd020b55f040 P 27aac7e335b4450da720babd10ed7809 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:42.294406 9974 raft_consensus.cc:2258] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:42.294443 9973 raft_consensus.cc:2229] T 519c7cb40eaa4e6495bacd020b55f040 P e867826874874526b87f5fef3a61a238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:43:42.294486 9973 raft_consensus.cc:2258] T 519c7cb40eaa4e6495bacd020b55f040 P e867826874874526b87f5fef3a61a238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:43:42.294768 9973 ts_tablet_manager.cc:1905] T 519c7cb40eaa4e6495bacd020b55f040 P e867826874874526b87f5fef3a61a238: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:42.294836 9974 ts_tablet_manager.cc:1905] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:42.296586 9974 ts_tablet_manager.cc:1918] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:43:42.296633 9972 ts_tablet_manager.cc:1905] T 519c7cb40eaa4e6495bacd020b55f040 P 27aac7e335b4450da720babd10ed7809: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:43:42.296665 9974 log.cc:1196] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-0/wal/wals/519c7cb40eaa4e6495bacd020b55f040
I20260928 04:43:42.296708 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 9507
I20260928 04:43:42.296916 9974 ts_tablet_manager.cc:1939] T 519c7cb40eaa4e6495bacd020b55f040 P fd16e3dcebe3400b8e6985dd10659cf5: Deleting consensus metadata
I20260928 04:43:42.298136 9973 ts_tablet_manager.cc:1918] T 519c7cb40eaa4e6495bacd020b55f040 P e867826874874526b87f5fef3a61a238: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:43:42.298209 9973 log.cc:1196] T 519c7cb40eaa4e6495bacd020b55f040 P e867826874874526b87f5fef3a61a238: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-2/wal/wals/519c7cb40eaa4e6495bacd020b55f040
I20260928 04:43:42.298447 9973 ts_tablet_manager.cc:1939] T 519c7cb40eaa4e6495bacd020b55f040 P e867826874874526b87f5fef3a61a238: Deleting consensus metadata
I20260928 04:43:42.298796 9421 catalog_manager.cc:4841] TS e867826874874526b87f5fef3a61a238 (127.1.226.195:34427): tablet 519c7cb40eaa4e6495bacd020b55f040 (table test-table [id=cf73d5bb4c6641a2b0fb1ebf0539c0c8]) successfully deleted
I20260928 04:43:42.300313 9972 ts_tablet_manager.cc:1918] T 519c7cb40eaa4e6495bacd020b55f040 P 27aac7e335b4450da720babd10ed7809: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:43:42.300383 9972 log.cc:1196] T 519c7cb40eaa4e6495bacd020b55f040 P 27aac7e335b4450da720babd10ed7809: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570585693787-1931-0/minicluster-data/ts-1/wal/wals/519c7cb40eaa4e6495bacd020b55f040
I20260928 04:43:42.300616 9972 ts_tablet_manager.cc:1939] T 519c7cb40eaa4e6495bacd020b55f040 P 27aac7e335b4450da720babd10ed7809: Deleting consensus metadata
I20260928 04:43:42.300995 9422 catalog_manager.cc:4841] TS 27aac7e335b4450da720babd10ed7809 (127.1.226.194:44519): tablet 519c7cb40eaa4e6495bacd020b55f040 (table test-table [id=cf73d5bb4c6641a2b0fb1ebf0539c0c8]) successfully deleted
W20260928 04:43:42.303941 9419 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.226.193:40603 (error 108)
I20260928 04:43:42.304247 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 9647
I20260928 04:43:42.304965 10036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:42.304335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:40603 (local address 127.0.0.1:39012)
0928 04:43:42.304857 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:42.304909 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.193:40603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":45,"threads_started":1}
W20260928 04:43:42.305804 9419 catalog_manager.cc:4584] TS fd16e3dcebe3400b8e6985dd10659cf5 (127.1.226.193:40603): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 519c7cb40eaa4e6495bacd020b55f040: Network error: Client connection negotiation failed: client connection to 127.1.226.193:40603: connect: Connection refused (error 111)
I20260928 04:43:42.311259 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 9787
I20260928 04:43:42.316331 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 9403
2026-09-28T04:43:42Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3272 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:43:42 dist-test-slave-j5c3 krb5kdc[10039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:43:42 dist-test-slave-j5c3 krb5kdc[10039](info): set up 2 sockets
Sep 28 04:43:42 dist-test-slave-j5c3 krb5kdc[10039](info): commencing operation
krb5kdc: starting...
W20260928 04:43:43.356861 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:43:43 dist-test-slave-j5c3 krb5kdc[10039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:43:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:43:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:43:43.479192 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:33985
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:46767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:33985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:43.561739 10052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:43.561975 10052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:43.562000 10052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:43.563153 10052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:43:43.563207 10052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:43.563222 10052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:43.563235 10052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:43:43.563248 10052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:43:43.564817 10052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:46767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:33985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:33985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:43.565033 10052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:43.565234 10052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:43.570508 10057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:43.570492 10058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:43.570485 10060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:43.570839 10052 server_base.cc:1034] running on GCE node
I20260928 04:43:43.571084 10052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:43.571395 10052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:43.572546 10052 hybrid_clock.cc:648] HybridClock initialized: now 1790570623572512 us; error 48 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T04:43:43Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1260 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:43:43 dist-test-slave-j5c3 krb5kdc[10065](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:43:43 dist-test-slave-j5c3 krb5kdc[10065](info): set up 2 sockets
Sep 28 04:43:43 dist-test-slave-j5c3 krb5kdc[10065](info): commencing operation
krb5kdc: starting...
W20260928 04:43:44.616863 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:43:44 dist-test-slave-j5c3 krb5kdc[10065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:43:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:43:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:43:44.740962 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:38133
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:40957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:38133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:44.824482 10078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:44.824719 10078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:44.824756 10078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:44.825917 10078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:43:44.825987 10078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:44.826011 10078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:44.826052 10078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:43:44.826074 10078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:43:44.827551 10078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:40957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:38133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:38133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:44.827829 10078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:44.828063 10078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:44.833076 10084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:44.833086 10083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:44.833089 10086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:44.833295 10078 server_base.cc:1034] running on GCE node
I20260928 04:43:44.833575 10078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:44.833873 10078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:44.835067 10078 hybrid_clock.cc:648] HybridClock initialized: now 1790570624835052 us; error 31 us; skew 500 ppm
Sep 28 04:43:44 dist-test-slave-j5c3 krb5kdc[10065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570624, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:44.843288 10078 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:43:44.843822 10078 webserver.cc:460] Webserver started at http://127.1.226.254:45985/ using document root <none> and password file <none>
I20260928 04:43:44.844075 10078 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:44.844146 10078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:44.844322 10078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:44.845438 10078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "afe488528c234451b041c712005d0d34"
format_stamp: "Formatted at 2026-09-28 04:43:44 on dist-test-slave-j5c3"
server_key: "85ee2bb62846d4adf6d0cc03734c778b"
server_key_iv: "96ae0b138f43ae84a921bf7d74e8172f"
server_key_version: "encryptionkey@0"
I20260928 04:43:44.845805 10078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "afe488528c234451b041c712005d0d34"
format_stamp: "Formatted at 2026-09-28 04:43:44 on dist-test-slave-j5c3"
server_key: "85ee2bb62846d4adf6d0cc03734c778b"
server_key_iv: "96ae0b138f43ae84a921bf7d74e8172f"
server_key_version: "encryptionkey@0"
I20260928 04:43:44.847172 10078 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:43:44.847960 10093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:44.848264 10078 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:44.848368 10078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "afe488528c234451b041c712005d0d34"
format_stamp: "Formatted at 2026-09-28 04:43:44 on dist-test-slave-j5c3"
server_key: "85ee2bb62846d4adf6d0cc03734c778b"
server_key_iv: "96ae0b138f43ae84a921bf7d74e8172f"
server_key_version: "encryptionkey@0"
I20260928 04:43:44.848452 10078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:44.855749 10078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:44.857875 10078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:44.858049 10078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:44.862417 10078 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:38133
I20260928 04:43:44.862466 10145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:38133 every 8 connection(s)
I20260928 04:43:44.862854 10078 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:43:44.863734 10146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:43:44.866231 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 10078
I20260928 04:43:44.866221 10146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34: Bootstrap starting.
I20260928 04:43:44.866354 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:43:44.866532 1931 external_mini_cluster.cc:1428] Setting key afc4019c026cfe87dcfae62959665da1
I20260928 04:43:44.866988 10146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34: Neither blocks nor log segments found. Creating new log.
I20260928 04:43:44.867348 10146 log.cc:824] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:44.868103 10146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34: No bootstrap required, opened a new log
I20260928 04:43:44.870476 10146 raft_consensus.cc:348] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe488528c234451b041c712005d0d34" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38133 } }
I20260928 04:43:44.870656 10146 raft_consensus.cc:374] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:43:44.870689 10146 raft_consensus.cc:729] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afe488528c234451b041c712005d0d34, State: Initialized, Role: FOLLOWER
I20260928 04:43:44.870841 10146 consensus_queue.cc:260] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe488528c234451b041c712005d0d34" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38133 } }
I20260928 04:43:44.870916 10146 raft_consensus.cc:388] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:43:44.870939 10146 raft_consensus.cc:482] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:43:44.871004 10146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:43:44.871642 10146 raft_consensus.cc:504] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe488528c234451b041c712005d0d34" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38133 } }
I20260928 04:43:44.871757 10146 leader_election.cc:302] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afe488528c234451b041c712005d0d34; no voters:
I20260928 04:43:44.871914 10146 leader_election.cc:288] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:43:44.871982 10151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:43:44.872131 10151 raft_consensus.cc:686] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [term 1 LEADER]: Becoming Leader. State: Replica: afe488528c234451b041c712005d0d34, State: Running, Role: LEADER
I20260928 04:43:44.872184 10146 sys_catalog.cc:564] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:43:44.873616 10151 consensus_queue.cc:237] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe488528c234451b041c712005d0d34" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38133 } }
Sep 28 04:43:44 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:44.874297 10152 sys_catalog.cc:455] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afe488528c234451b041c712005d0d34. Latest consensus state: current_term: 1 leader_uuid: "afe488528c234451b041c712005d0d34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe488528c234451b041c712005d0d34" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38133 } } }
I20260928 04:43:44.874395 10153 sys_catalog.cc:455] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afe488528c234451b041c712005d0d34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe488528c234451b041c712005d0d34" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38133 } } }
I20260928 04:43:44.874847 10153 sys_catalog.cc:458] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:44.874845 10152 sys_catalog.cc:458] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [sys.catalog]: This master's current role is: LEADER
W20260928 04:43:44.875548 10165 catalog_manager.cc:1558] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:43:44.875605 10165 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:43:44.875680 10167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:43:44.875849 10167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:43:44.877573 10167 catalog_manager.cc:1347] Generated new cluster ID: 31ba85422bdb43869d221c4d5bce4367
I20260928 04:43:44.877631 10167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:43:44.879356 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:44.867121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33498 (local address 127.1.226.254:38133)
0928 04:43:44.867338 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:44.867348 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:44.867382 (+ 34us) server_negotiation.cc:407] Connection header received
0928 04:43:44.868029 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:44.868038 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:44.868101 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:44.868295 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:44.868945 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:44.870461 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:44.871450 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:44.872087 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:44.872564 (+ 477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:44.875080 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:44.875086 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:44.875086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:44.875100 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:44.877319 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:44.877768 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:44.877771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:44.877772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:44.877878 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:44.878062 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:44.878063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:44.878063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:44.878362 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:44.878483 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:44.878993 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 04:43:44.879125 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":97,"threads_started":1}
I20260928 04:43:44.887074 10167 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:43:44.887607 10167 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:43:44.896062 10167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34: Generated new TSK 0
I20260928 04:43:44.896329 10167 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.226.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
I20260928 04:43:44.933282 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:0
--local_ip_for_outbound_sockets=127.1.226.193
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:38133
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:40957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:45.041491 10172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:45.041719 10172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:45.041744 10172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:45.042857 10172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:45.042891 10172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:45.042917 10172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:43:45.044282 10172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:40957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:38133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:45.044461 10172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:45.044631 10172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:43:45.049597 10172 server_base.cc:1034] running on GCE node
W20260928 04:43:45.049566 10177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:45.049551 10180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:45.049602 10178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:45.050086 10172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:45.050288 10172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:45.051429 10172 hybrid_clock.cc:648] HybridClock initialized: now 1790570625051409 us; error 39 us; skew 500 ppm
Sep 28 04:43:45 dist-test-slave-j5c3 krb5kdc[10065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570625, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:45.054134 10172 init.cc:377] Logged in from keytab as kudu/127.1.226.193@KRBTEST.COM (short username kudu)
I20260928 04:43:45.054693 10172 webserver.cc:460] Webserver started at http://127.1.226.193:42791/ using document root <none> and password file <none>
I20260928 04:43:45.054944 10172 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:45.055016 10172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:45.055179 10172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:45.056720 10172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/data/instance:
uuid: "f9a251ccb8d347dc8fd7a6a750d388a3"
format_stamp: "Formatted at 2026-09-28 04:43:45 on dist-test-slave-j5c3"
server_key: "aa3517bf82c42e0d2388aa42630756fd"
server_key_iv: "b786265392be732348e6ddb585f68511"
server_key_version: "encryptionkey@0"
I20260928 04:43:45.057194 10172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance:
uuid: "f9a251ccb8d347dc8fd7a6a750d388a3"
format_stamp: "Formatted at 2026-09-28 04:43:45 on dist-test-slave-j5c3"
server_key: "aa3517bf82c42e0d2388aa42630756fd"
server_key_iv: "b786265392be732348e6ddb585f68511"
server_key_version: "encryptionkey@0"
I20260928 04:43:45.058897 10172 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:43:45.059795 10187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:45.059967 10172 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:45.060072 10172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "f9a251ccb8d347dc8fd7a6a750d388a3"
format_stamp: "Formatted at 2026-09-28 04:43:45 on dist-test-slave-j5c3"
server_key: "aa3517bf82c42e0d2388aa42630756fd"
server_key_iv: "b786265392be732348e6ddb585f68511"
server_key_version: "encryptionkey@0"
I20260928 04:43:45.060200 10172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:45.078240 10172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:45.080622 10172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:45.080811 10172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:45.081060 10172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:45.081394 10172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:45.081449 10172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:45.081490 10172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:45.081518 10172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:45.087579 10172 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:35627
I20260928 04:43:45.087632 10300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:35627 every 8 connection(s)
I20260928 04:43:45.087975 10172 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
I20260928 04:43:45.089322 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 10172
I20260928 04:43:45.089439 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance
I20260928 04:43:45.089684 1931 external_mini_cluster.cc:1428] Setting key 801f3d95a8ee042709a28068492d7cd7
Sep 28 04:43:45 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570625, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.226.194@KRBTEST.COM; defaulting to no policy
I20260928 04:43:45.098682 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:45.088786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:40827 (local address 127.1.226.254:38133)
0928 04:43:45.088850 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:45.088852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:45.089089 (+ 237us) server_negotiation.cc:407] Connection header received
0928 04:43:45.089673 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:45.089674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:45.089695 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:45.089748 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:45.090939 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.091443 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.092531 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.092904 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.092978 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:45.095304 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:45.095306 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:45.095306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:45.095341 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:45.097215 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:45.097642 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:45.097643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:45.097644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:45.097706 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:45.097968 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:45.097969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:45.097969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:45.098119 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:45.098198 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:45.098472 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 04:43:45.098572 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 04:43:45.098860 10303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:45.088740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.193:40827)
0928 04:43:45.088924 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:45.088968 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:45.089556 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:45.089828 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:45.089832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:45.090629 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:45.090835 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:45.090839 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.091539 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.091540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:45.092450 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:45.092452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.093013 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.093014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:45.093128 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:45.093800 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:45.093808 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:45.095212 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:45.097304 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:45.097310 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:45.097312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:45.097539 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:45.097784 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:45.097784 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:45.097785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:45.097888 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:45.098188 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:45.098192 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:45.098339 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 04:43:45.098445 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 04:43:45.098573 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":108,"threads_started":1}
I20260928 04:43:45.099581 10301 heartbeater.cc:347] Connected to a master server at 127.1.226.254:38133
I20260928 04:43:45.099678 10301 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:45.099886 10301 heartbeater.cc:511] Master 127.1.226.254:38133 requested a full tablet report, sending...
I20260928 04:43:45.100489 10110 ts_manager.cc:194] Registered new tserver with Master: f9a251ccb8d347dc8fd7a6a750d388a3 (127.1.226.193:35627)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.194@KRBTEST.COM" created.
I20260928 04:43:45.101182 10110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.193@KRBTEST.COM'} at 127.1.226.193:40827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
I20260928 04:43:45.115511 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:0
--local_ip_for_outbound_sockets=127.1.226.194
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:38133
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:40957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:45.197317 10306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:45.197526 10306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:45.197551 10306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:45.198791 10306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:45.198835 10306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:45.198863 10306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:43:45.200371 10306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:40957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:38133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:45.200613 10306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:45.200848 10306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:45.206243 10314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:45.206244 10312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:45.206259 10311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:45.206249 10306 server_base.cc:1034] running on GCE node
I20260928 04:43:45.206759 10306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:45.207000 10306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:45.208204 10306 hybrid_clock.cc:648] HybridClock initialized: now 1790570625208115 us; error 59 us; skew 500 ppm
Sep 28 04:43:45 dist-test-slave-j5c3 krb5kdc[10065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570625, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:45.210950 10306 init.cc:377] Logged in from keytab as kudu/127.1.226.194@KRBTEST.COM (short username kudu)
I20260928 04:43:45.211467 10306 webserver.cc:460] Webserver started at http://127.1.226.194:34265/ using document root <none> and password file <none>
I20260928 04:43:45.211709 10306 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:45.211755 10306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:45.211899 10306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:45.213074 10306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/data/instance:
uuid: "491c6c8d223045b092c7cfcc978c440e"
format_stamp: "Formatted at 2026-09-28 04:43:45 on dist-test-slave-j5c3"
server_key: "4020d803c27cc053e9695853804526c3"
server_key_iv: "f9f7dc95b42d24a0823c714503076e94"
server_key_version: "encryptionkey@0"
I20260928 04:43:45.213442 10306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance:
uuid: "491c6c8d223045b092c7cfcc978c440e"
format_stamp: "Formatted at 2026-09-28 04:43:45 on dist-test-slave-j5c3"
server_key: "4020d803c27cc053e9695853804526c3"
server_key_iv: "f9f7dc95b42d24a0823c714503076e94"
server_key_version: "encryptionkey@0"
I20260928 04:43:45.214861 10306 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:43:45.215698 10321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:45.215970 10306 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:43:45.216078 10306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "491c6c8d223045b092c7cfcc978c440e"
format_stamp: "Formatted at 2026-09-28 04:43:45 on dist-test-slave-j5c3"
server_key: "4020d803c27cc053e9695853804526c3"
server_key_iv: "f9f7dc95b42d24a0823c714503076e94"
server_key_version: "encryptionkey@0"
I20260928 04:43:45.216192 10306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:45.231398 10306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:45.233673 10306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:45.233867 10306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:45.234124 10306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:45.234462 10306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:45.234515 10306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:45.234555 10306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:45.234583 10306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:45.241077 10306 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:33519
I20260928 04:43:45.241129 10434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:33519 every 8 connection(s)
I20260928 04:43:45.241494 10306 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
Sep 28 04:43:45 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570625, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:45.250792 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 10306
I20260928 04:43:45.250923 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance
I20260928 04:43:45.251080 1931 external_mini_cluster.cc:1428] Setting key 6a0af229e856ea79c3437279aa6f0ce9
I20260928 04:43:45.251678 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:45.242318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:44999 (local address 127.1.226.254:38133)
0928 04:43:45.242365 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:45.242367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:45.242733 (+ 366us) server_negotiation.cc:407] Connection header received
0928 04:43:45.243216 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:45.243218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:45.243228 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:45.243292 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:45.244389 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.244882 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.246022 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.246256 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.246325 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:45.248631 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:45.248633 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:45.248633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:45.248641 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:45.250022 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:45.250436 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:45.250438 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:45.250439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:45.250501 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:45.250885 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:45.250886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:45.250887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:45.251048 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:45.251150 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:45.251485 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 04:43:45.251599 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:43:45.251912 10437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:45.242297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.194:44999)
0928 04:43:45.242485 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:45.242532 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:45.243114 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:45.243351 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:45.243355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:45.243986 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:45.244292 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:45.244306 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.244971 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.244972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:45.245926 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:45.245928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.246365 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.246366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:45.246477 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:45.246958 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:45.246968 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:45.248512 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:45.250104 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:45.250110 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:45.250112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:45.250337 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:45.250573 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:45.250573 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:45.250574 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:45.250749 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:45.251139 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:45.251146 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:45.251319 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 04:43:45.251420 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 04:43:45.251608 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":101,"threads_started":1}
I20260928 04:43:45.252429 10435 heartbeater.cc:347] Connected to a master server at 127.1.226.254:38133
I20260928 04:43:45.252555 10435 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:45.252795 10435 heartbeater.cc:511] Master 127.1.226.254:38133 requested a full tablet report, sending...
I20260928 04:43:45.253189 10110 ts_manager.cc:194] Registered new tserver with Master: 491c6c8d223045b092c7cfcc978c440e (127.1.226.194:33519)
I20260928 04:43:45.253583 10110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.194@KRBTEST.COM'} at 127.1.226.194:44999
WARNING: no policy specified for kudu/127.1.226.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
I20260928 04:43:45.279086 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:0
--local_ip_for_outbound_sockets=127.1.226.195
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:38133
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:40957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:45.361369 10440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:45.361614 10440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:45.361652 10440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:45.362886 10440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:45.362947 10440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:45.362993 10440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:43:45.364547 10440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:40957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:38133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:45.364817 10440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:45.365043 10440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:45.370363 10446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:45.370431 10448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:45.370600 10440 server_base.cc:1034] running on GCE node
W20260928 04:43:45.370555 10445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:45.370956 10440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:45.371158 10440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:45.372306 10440 hybrid_clock.cc:648] HybridClock initialized: now 1790570625372295 us; error 31 us; skew 500 ppm
Sep 28 04:43:45 dist-test-slave-j5c3 krb5kdc[10065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570625, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:45.375352 10440 init.cc:377] Logged in from keytab as kudu/127.1.226.195@KRBTEST.COM (short username kudu)
I20260928 04:43:45.375820 10440 webserver.cc:460] Webserver started at http://127.1.226.195:32801/ using document root <none> and password file <none>
I20260928 04:43:45.376073 10440 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:45.376129 10440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:45.376338 10440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:43:45.377508 10440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/data/instance:
uuid: "71e9b1b1f48b41649613476a1f268ef2"
format_stamp: "Formatted at 2026-09-28 04:43:45 on dist-test-slave-j5c3"
server_key: "511e4cb35cfe332ce3a7c00aaa95d9c9"
server_key_iv: "bf829b67091ce42054b9260cfd0a6a54"
server_key_version: "encryptionkey@0"
I20260928 04:43:45.377905 10440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance:
uuid: "71e9b1b1f48b41649613476a1f268ef2"
format_stamp: "Formatted at 2026-09-28 04:43:45 on dist-test-slave-j5c3"
server_key: "511e4cb35cfe332ce3a7c00aaa95d9c9"
server_key_iv: "bf829b67091ce42054b9260cfd0a6a54"
server_key_version: "encryptionkey@0"
I20260928 04:43:45.379485 10440 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:43:45.380379 10455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:45.380596 10440 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:45.380735 10440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "71e9b1b1f48b41649613476a1f268ef2"
format_stamp: "Formatted at 2026-09-28 04:43:45 on dist-test-slave-j5c3"
server_key: "511e4cb35cfe332ce3a7c00aaa95d9c9"
server_key_iv: "bf829b67091ce42054b9260cfd0a6a54"
server_key_version: "encryptionkey@0"
I20260928 04:43:45.380808 10440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:45.396335 10440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:45.398667 10440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:45.398837 10440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:45.399060 10440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:45.399395 10440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:45.399451 10440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:45.399498 10440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:45.399537 10440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:45.405665 10440 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:36197
I20260928 04:43:45.405776 10568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:36197 every 8 connection(s)
I20260928 04:43:45.406066 10440 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
Sep 28 04:43:45 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570625, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:43:45.414290 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 10440
I20260928 04:43:45.414381 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance
I20260928 04:43:45.414601 1931 external_mini_cluster.cc:1428] Setting key 7b34669976d41906c98dea2080bff3e3
I20260928 04:43:45.416388 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:45.406875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:50855 (local address 127.1.226.254:38133)
0928 04:43:45.406959 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:45.406962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:45.407166 (+ 204us) server_negotiation.cc:407] Connection header received
0928 04:43:45.407767 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:45.407769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:45.407782 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:45.407827 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:45.408985 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.409551 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.410662 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.410882 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.410965 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:45.413285 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:45.413286 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:45.413287 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:45.413294 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:45.414832 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:45.415467 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:45.415467 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:45.415468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:45.415548 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:45.415807 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:45.415808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:45.415808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:45.415920 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:45.415973 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:45.416258 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 04:43:45.416323 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 04:43:45.416599 10571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:45.406831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.195:50855)
0928 04:43:45.407017 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:45.407068 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:45.407622 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:45.407920 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:45.407926 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:45.408660 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:45.408866 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:45.408871 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.409674 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.409675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:45.410555 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:45.410557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.410970 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.410971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:45.411106 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:45.411675 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:45.411685 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:45.413195 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:43:45.414932 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:45.414939 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:45.414941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:45.415264 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:45.415619 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:43:45.415620 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:43:45.415620 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:43:45.415746 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:43:45.415976 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:43:45.415981 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:43:45.416094 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 04:43:45.416233 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 04:43:45.416375 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":84,"threads_started":1}
I20260928 04:43:45.417069 10569 heartbeater.cc:347] Connected to a master server at 127.1.226.254:38133
I20260928 04:43:45.417150 10569 heartbeater.cc:464] Registering TS with master...
I20260928 04:43:45.417292 10569 heartbeater.cc:511] Master 127.1.226.254:38133 requested a full tablet report, sending...
I20260928 04:43:45.417615 10110 ts_manager.cc:194] Registered new tserver with Master: 71e9b1b1f48b41649613476a1f268ef2 (127.1.226.195:36197)
I20260928 04:43:45.418006 10110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.195@KRBTEST.COM'} at 127.1.226.195:50855
I20260928 04:43:45.426671 1931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.226.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.226.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.226.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.193.keytab.
Entry for principal oryx/127.1.226.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.193.keytab.
WARNING: no policy specified for oryx/127.1.226.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.226.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.226.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.194.keytab.
Entry for principal oryx/127.1.226.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.194.keytab.
WARNING: no policy specified for oryx/127.1.226.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.226.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.226.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.195.keytab.
Entry for principal oryx/127.1.226.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.195.keytab.
I20260928 04:43:45.503396 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 10172
I20260928 04:43:45.508761 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 10306
I20260928 04:43:45.513535 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 10440
I20260928 04:43:45.518392 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 10078
I20260928 04:43:45.523396 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:38133
--webserver_interface=127.1.226.254
--webserver_port=45985
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:40957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:38133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:45.605979 10578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:45.606192 10578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:45.606215 10578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:45.607398 10578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:43:45.607440 10578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:45.607455 10578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:45.607496 10578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:43:45.607525 10578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:43:45.609124 10578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:40957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:38133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:38133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=45985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:45.609401 10578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:45.609642 10578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:45.614437 10584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:45.614444 10583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:45.614445 10586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:45.614799 10578 server_base.cc:1034] running on GCE node
I20260928 04:43:45.615060 10578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:45.615357 10578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:45.616559 10578 hybrid_clock.cc:648] HybridClock initialized: now 1790570625616485 us; error 60 us; skew 500 ppm
Sep 28 04:43:45 dist-test-slave-j5c3 krb5kdc[10065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570625, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:45.620103 10578 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:43:45.620627 10578 webserver.cc:460] Webserver started at http://127.1.226.254:45985/ using document root <none> and password file <none>
I20260928 04:43:45.620945 10578 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:45.621057 10578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:45.623247 10578 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:43:45.628405 10593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:45.628731 10578 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.000s
I20260928 04:43:45.628865 10578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "afe488528c234451b041c712005d0d34"
format_stamp: "Formatted at 2026-09-28 04:43:44 on dist-test-slave-j5c3"
server_key: "85ee2bb62846d4adf6d0cc03734c778b"
server_key_iv: "96ae0b138f43ae84a921bf7d74e8172f"
server_key_version: "encryptionkey@0"
I20260928 04:43:45.629454 10578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:45.659644 10578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:45.662884 10578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:45.663057 10578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:45.667687 10578 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:38133
I20260928 04:43:45.667752 10645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:38133 every 8 connection(s)
I20260928 04:43:45.668136 10578 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:43:45.668751 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 10578
I20260928 04:43:45.669135 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:35627
--local_ip_for_outbound_sockets=127.1.226.193
--tserver_master_addrs=127.1.226.254:38133
--webserver_port=42791
--webserver_interface=127.1.226.193
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:40957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.226.193
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.193.keytab with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:43:45.670861 10646 sys_catalog.cc:263] Verifying existing consensus state
I20260928 04:43:45.671566 10646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34: Bootstrap starting.
I20260928 04:43:45.680963 10646 log.cc:824] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34: Log is configured to *not* fsync() on all Append() calls
I20260928 04:43:45.682894 10646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:43:45.683207 10646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34: Bootstrap complete.
I20260928 04:43:45.685678 10646 raft_consensus.cc:348] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe488528c234451b041c712005d0d34" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38133 } }
I20260928 04:43:45.685997 10646 raft_consensus.cc:729] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: afe488528c234451b041c712005d0d34, State: Initialized, Role: FOLLOWER
I20260928 04:43:45.686195 10646 consensus_queue.cc:260] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe488528c234451b041c712005d0d34" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38133 } }
I20260928 04:43:45.686311 10646 raft_consensus.cc:388] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:43:45.686379 10646 raft_consensus.cc:482] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:43:45.686429 10646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [term 1 FOLLOWER]: Advancing to term 2
I20260928 04:43:45.687714 10646 raft_consensus.cc:504] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe488528c234451b041c712005d0d34" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38133 } }
I20260928 04:43:45.687839 10646 leader_election.cc:302] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afe488528c234451b041c712005d0d34; no voters:
I20260928 04:43:45.688063 10646 leader_election.cc:288] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 04:43:45.688104 10650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 04:43:45.688261 10650 raft_consensus.cc:686] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [term 2 LEADER]: Becoming Leader. State: Replica: afe488528c234451b041c712005d0d34, State: Running, Role: LEADER
I20260928 04:43:45.688376 10646 sys_catalog.cc:564] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:43:45.688354 10650 consensus_queue.cc:237] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe488528c234451b041c712005d0d34" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38133 } }
I20260928 04:43:45.688683 10650 sys_catalog.cc:455] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afe488528c234451b041c712005d0d34. Latest consensus state: current_term: 2 leader_uuid: "afe488528c234451b041c712005d0d34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe488528c234451b041c712005d0d34" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38133 } } }
I20260928 04:43:45.688764 10650 sys_catalog.cc:458] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:45.688685 10651 sys_catalog.cc:455] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "afe488528c234451b041c712005d0d34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe488528c234451b041c712005d0d34" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38133 } } }
I20260928 04:43:45.688941 10651 sys_catalog.cc:458] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34 [sys.catalog]: This master's current role is: LEADER
I20260928 04:43:45.688975 10655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:43:45.689553 10655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:43:45.689754 10655 catalog_manager.cc:1259] Loaded cluster ID: 31ba85422bdb43869d221c4d5bce4367
I20260928 04:43:45.690022 10655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:43:45.690930 10655 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:43:45.691140 10655 catalog_manager.cc:5879] T 00000000000000000000000000000000 P afe488528c234451b041c712005d0d34: Loaded TSK: 0
I20260928 04:43:45.691366 10655 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 04:43:45.780223 10648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:45.780534 10648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:45.780570 10648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:45.782284 10648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:45.782366 10648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:45.782409 10648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:43:45.784585 10648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:40957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.193.keytab
--principal=oryx/127.1.226.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:35627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=42791
--tserver_master_addrs=127.1.226.254:38133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:45.784852 10648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:45.785126 10648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:45.790952 10675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:45.791056 10673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:45.790994 10672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:45.791419 10648 server_base.cc:1034] running on GCE node
I20260928 04:43:45.791639 10648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:45.791910 10648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:45.793093 10648 hybrid_clock.cc:648] HybridClock initialized: now 1790570625793080 us; error 35 us; skew 500 ppm
Sep 28 04:43:45 dist-test-slave-j5c3 krb5kdc[10065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570625, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:45.796121 10648 init.cc:377] Logged in from keytab as oryx/127.1.226.193@KRBTEST.COM (short username oryx)
I20260928 04:43:45.796653 10648 webserver.cc:460] Webserver started at http://127.1.226.193:42791/ using document root <none> and password file <none>
I20260928 04:43:45.796959 10648 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:45.797040 10648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:45.798571 10648 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:43:45.799319 10682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:45.799533 10648 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:45.799625 10648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "f9a251ccb8d347dc8fd7a6a750d388a3"
format_stamp: "Formatted at 2026-09-28 04:43:45 on dist-test-slave-j5c3"
server_key: "aa3517bf82c42e0d2388aa42630756fd"
server_key_iv: "b786265392be732348e6ddb585f68511"
server_key_version: "encryptionkey@0"
I20260928 04:43:45.800078 10648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:45.831308 10648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:45.834100 10648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:45.834256 10648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:45.834545 10648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:45.834921 10648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:45.834971 10648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:45.835003 10648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:45.835021 10648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:45.842361 10648 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:35627
I20260928 04:43:45.842455 10795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:35627 every 8 connection(s)
I20260928 04:43:45.842815 10648 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
I20260928 04:43:45.848384 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 10648
I20260928 04:43:45.848747 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:33519
--local_ip_for_outbound_sockets=127.1.226.194
--tserver_master_addrs=127.1.226.254:38133
--webserver_port=34265
--webserver_interface=127.1.226.194
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:40957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.226.194
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.194.keytab with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 04:43:45 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:45 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:45.853184 10798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:45.843770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.193:57881)
0928 04:43:45.843956 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:45.843998 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:45.844543 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:45.845009 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:45.845012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:45.845611 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:45.845825 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:45.845829 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.847436 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.847437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:45.848389 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:45.848392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.849292 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.849293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:45.849458 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:45.850093 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:45.850105 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:45.852973 (+ 2868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":104,"threads_started":1}
W20260928 04:43:45.853457 10796 heartbeater.cc:644] Failed to heartbeat to 127.1.226.254:38133 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.226.254:38133: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
W20260928 04:43:45.853716 10799 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:45.843864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:57881 (local address 127.1.226.254:38133)
0928 04:43:45.844311 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:45.844320 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:45.844372 (+ 52us) server_negotiation.cc:407] Connection header received
0928 04:43:45.844742 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:45.844750 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:45.844806 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:45.844964 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:45.845956 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.847328 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.848552 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.849187 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.849313 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:45.853165 (+ 3852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.193:57881: BlockingRecv error: recv got EOF from 127.1.226.193:57881 (error 108)
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":95,"threads_started":1}
Sep 28 04:43:45 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:45 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:45.859934 10798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:45.853723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.193:47679)
0928 04:43:45.853957 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:45.853968 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:45.854050 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:45.854220 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:45.854221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:45.854395 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:45.854462 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:45.854464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.855499 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.855500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:45.856380 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:45.856384 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.856927 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.856928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:45.857005 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:45.857567 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:45.857572 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:45.859853 (+ 2281us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216}
W20260928 04:43:45.860292 10799 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:45.853813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:47679 (local address 127.1.226.254:38133)
0928 04:43:45.853939 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:45.853941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:45.854033 (+ 92us) server_negotiation.cc:407] Connection header received
0928 04:43:45.854113 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:45.854114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:45.854128 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:45.854190 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:45.854556 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.855394 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.856490 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.856834 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.856913 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:45.860219 (+ 3306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.193:47679: BlockingRecv error: recv got EOF from 127.1.226.193:47679 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":106,"threads_started":1}
Sep 28 04:43:45 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:45 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:45.866050 10798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:45.860235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.193:44241)
0928 04:43:45.860341 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:45.860353 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:45.860412 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:45.860603 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:45.860604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:45.860753 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:45.860793 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:45.860795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.861912 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.861913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:45.862703 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:45.862705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.863223 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.863224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:45.863280 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:45.863780 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:45.863786 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:45.865895 (+ 2109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87}
W20260928 04:43:45.866194 10796 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 04:43:45.866312 10801 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:45.860293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:44241 (local address 127.1.226.254:38133)
0928 04:43:45.860346 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:45.860348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:45.860393 (+ 45us) server_negotiation.cc:407] Connection header received
0928 04:43:45.860474 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:45.860475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:45.860486 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:45.860556 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:45.860888 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.861811 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.862795 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.863138 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.863213 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:45.866025 (+ 2812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.193:44241: BlockingRecv error: recv got EOF from 127.1.226.193:44241 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 04:43:45.936092 10800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:45.936316 10800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:45.936339 10800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:45.937533 10800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:45.937577 10800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:45.937602 10800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:43:45.939028 10800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:40957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.194.keytab
--principal=oryx/127.1.226.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:33519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=34265
--tserver_master_addrs=127.1.226.254:38133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:45.939267 10800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:45.939515 10800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:45.944707 10806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:45.944792 10807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:45.944707 10809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:45.944723 10800 server_base.cc:1034] running on GCE node
I20260928 04:43:45.945088 10800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:45.945369 10800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:45.946544 10800 hybrid_clock.cc:648] HybridClock initialized: now 1790570625946518 us; error 45 us; skew 500 ppm
Sep 28 04:43:45 dist-test-slave-j5c3 krb5kdc[10065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570625, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:45.949294 10800 init.cc:377] Logged in from keytab as oryx/127.1.226.194@KRBTEST.COM (short username oryx)
I20260928 04:43:45.949857 10800 webserver.cc:460] Webserver started at http://127.1.226.194:34265/ using document root <none> and password file <none>
I20260928 04:43:45.950106 10800 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:45.950204 10800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:45.951607 10800 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:43:45.952383 10816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:45.952589 10800 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:43:45.952689 10800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "491c6c8d223045b092c7cfcc978c440e"
format_stamp: "Formatted at 2026-09-28 04:43:45 on dist-test-slave-j5c3"
server_key: "4020d803c27cc053e9695853804526c3"
server_key_iv: "f9f7dc95b42d24a0823c714503076e94"
server_key_version: "encryptionkey@0"
I20260928 04:43:45.953066 10800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:45.978577 10800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:45.981089 10800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:45.981271 10800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:45.981516 10800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:45.981855 10800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:45.981909 10800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:45.981945 10800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:45.981973 10800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:45.987996 10800 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:33519
I20260928 04:43:45.988046 10929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:33519 every 8 connection(s)
I20260928 04:43:45.988490 10800 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
Sep 28 04:43:45 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:45.995411 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 10800
Sep 28 04:43:45 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:45.995807 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:36197
--local_ip_for_outbound_sockets=127.1.226.195
--tserver_master_addrs=127.1.226.254:38133
--webserver_port=32801
--webserver_interface=127.1.226.195
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:40957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.226.195
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.195.keytab with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:43:45.996762 10801 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:45.989345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:56051 (local address 127.1.226.254:38133)
0928 04:43:45.989474 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:45.989477 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:45.989627 (+ 150us) server_negotiation.cc:407] Connection header received
0928 04:43:45.990131 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:45.990133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:45.990144 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:45.990220 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:45.991236 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.991748 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.992905 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.993137 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.993213 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:45.996712 (+ 3499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.194:56051: BlockingRecv error: recv got EOF from 127.1.226.194:56051 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 04:43:45.996803 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:45.989248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.194:56051)
0928 04:43:45.989489 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:45.989542 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:45.990020 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:45.990306 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:45.990310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:45.990933 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:45.991131 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:45.991136 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.991861 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.991862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:45.992804 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:45.992808 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.993230 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.993231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:45.993355 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:45.993872 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:45.993881 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:45.996567 (+ 2686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":83,"threads_started":1}
W20260928 04:43:45.997444 10930 heartbeater.cc:644] Failed to heartbeat to 127.1.226.254:38133 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.226.254:38133: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Sep 28 04:43:45 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:45 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:46.002446 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:45.997764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.194:55939)
0928 04:43:45.997842 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:45.997868 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:45.997912 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:45.998038 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:45.998039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:45.998174 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:45.998224 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:45.998226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.998901 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.998902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:45.999490 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:45.999492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.999874 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.999875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:45.999949 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:46.000445 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:46.000450 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:46.002340 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 04:43:46.002524 10801 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:45.997827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:55939 (local address 127.1.226.254:38133)
0928 04:43:45.997888 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:45.997890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:45.997898 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:43:45.997975 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:45.997975 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:45.997987 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:45.998037 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:45.998310 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.998819 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.999589 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:45.999790 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:45.999873 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:46.002448 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.194:55939: BlockingRecv error: recv got EOF from 127.1.226.194:55939 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 04:43:46 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:46 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:46.008431 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:46.002805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.194:59993)
0928 04:43:46.002851 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:46.002861 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:46.002894 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:46.003057 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:46.003058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:46.003197 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:46.003226 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:46.003227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:46.004286 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:46.004288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:46.005299 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:46.005302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:46.005828 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:46.005829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:46.005897 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:46.006328 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:46.006334 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:46.008291 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 04:43:46.008482 10801 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:46.002883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:59993 (local address 127.1.226.254:38133)
0928 04:43:46.002925 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:46.002927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:46.002936 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:43:46.002961 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:46.002962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:46.002975 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:46.003047 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:46.003302 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:46.004185 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:46.005419 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:46.005744 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:46.005808 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:46.008416 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.194:59993: BlockingRecv error: recv got EOF from 127.1.226.194:59993 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 04:43:46.008569 10930 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 04:43:46.082149 10933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:43:46.082418 10933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:43:46.082456 10933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:43:46.083696 10933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:43:46.083768 10933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:43:46.083815 10933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:43:46.085316 10933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:40957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/krb5kdc/oryx_127.1.226.195.keytab
--principal=oryx/127.1.226.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:36197
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=32801
--tserver_master_addrs=127.1.226.254:38133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:43:46.085604 10933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:43:46.085850 10933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:43:46.091238 10939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:46.091256 10938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:43:46.091253 10941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:43:46.091486 10933 server_base.cc:1034] running on GCE node
I20260928 04:43:46.091711 10933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:43:46.092001 10933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:43:46.093174 10933 hybrid_clock.cc:648] HybridClock initialized: now 1790570626093158 us; error 37 us; skew 500 ppm
Sep 28 04:43:46 dist-test-slave-j5c3 krb5kdc[10065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570626, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.226.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:43:46.095984 10933 init.cc:377] Logged in from keytab as oryx/127.1.226.195@KRBTEST.COM (short username oryx)
I20260928 04:43:46.096619 10933 webserver.cc:460] Webserver started at http://127.1.226.195:32801/ using document root <none> and password file <none>
I20260928 04:43:46.096879 10933 fs_manager.cc:359] Metadata directory not provided
I20260928 04:43:46.096971 10933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:43:46.098569 10933 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:43:46.099331 10948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:46.099545 10933 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:43:46.099663 10933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "71e9b1b1f48b41649613476a1f268ef2"
format_stamp: "Formatted at 2026-09-28 04:43:45 on dist-test-slave-j5c3"
server_key: "511e4cb35cfe332ce3a7c00aaa95d9c9"
server_key_iv: "bf829b67091ce42054b9260cfd0a6a54"
server_key_version: "encryptionkey@0"
I20260928 04:43:46.100112 10933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:43:46.110673 10933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:43:46.112995 10933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:43:46.113183 10933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:43:46.113447 10933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:43:46.113797 10933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:43:46.113857 10933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:46.113895 10933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:43:46.113950 10933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:43:46.120599 10933 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:36197
I20260928 04:43:46.120651 11061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:36197 every 8 connection(s)
I20260928 04:43:46.121043 10933 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
I20260928 04:43:46.121348 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 10933
Sep 28 04:43:46 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:46 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:46.129796 10801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:46.121886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58034 (local address 127.1.226.254:38133)
0928 04:43:46.121940 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:46.121942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:46.122130 (+ 188us) server_negotiation.cc:407] Connection header received
0928 04:43:46.122294 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:46.122295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:46.122308 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:46.122383 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:46.123154 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:46.123763 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:46.124747 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:46.124936 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:46.125020 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:46.126324 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:43:46.126328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:43:46.126329 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:43:46.126347 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:43:46.128381 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:46.128750 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:46.128753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:46.128753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:46.128850 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:43:46.129059 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:43:46.129060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:43:46.129060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:43:46.129300 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:43:46.129448 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 04:43:46.129574 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 04:43:46.129697 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 04:43:46.129958 10799 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:46.122031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:57599 (local address 127.1.226.254:38133)
0928 04:43:46.122096 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:46.122098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:46.122279 (+ 181us) server_negotiation.cc:407] Connection header received
0928 04:43:46.123068 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:46.123069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:46.123081 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:46.123165 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:46.124259 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:46.124791 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:46.126104 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:46.126320 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:46.126412 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:46.129907 (+ 3495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.195:57599: BlockingRecv error: recv got EOF from 127.1.226.195:57599 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 04:43:46.129974 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:46.121965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.195:57599)
0928 04:43:46.122135 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:46.122185 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:46.122954 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:46.123240 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:46.123243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:46.123809 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:46.124112 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:46.124118 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:46.124909 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:46.124910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:46.125987 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:46.125990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:46.126425 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:46.126426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:46.126557 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:46.127153 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:46.127164 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:46.129746 (+ 2582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
W20260928 04:43:46.130214 11062 heartbeater.cc:644] Failed to heartbeat to 127.1.226.254:38133 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.226.254:38133: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Sep 28 04:43:46 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:46 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:46.135094 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:46.130448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.195:38701)
0928 04:43:46.130486 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:46.130498 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:46.130543 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:46.130733 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:46.130734 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:46.130861 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:46.130907 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:46.130908 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:46.131605 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:46.131606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:46.132219 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:46.132226 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:46.132646 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:46.132647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:46.132725 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:46.133167 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:46.133178 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:46.135001 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":20}
W20260928 04:43:46.135152 10801 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:46.130539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:38701 (local address 127.1.226.254:38133)
0928 04:43:46.130594 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:46.130596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:46.130607 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:43:46.130631 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:46.130632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:46.130640 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:46.130706 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:46.130998 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:46.131503 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:46.132323 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:46.132555 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:46.132646 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:46.135083 (+ 2437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.195:38701: BlockingRecv error: recv got EOF from 127.1.226.195:38701 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 04:43:46 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:46 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:46.139696 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:46.135354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.195:54187)
0928 04:43:46.135409 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:46.135421 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:46.135464 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:46.135626 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:46.135627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:46.135741 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:46.135770 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:46.135771 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:46.136563 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:46.136564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:46.137061 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:46.137062 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:46.137448 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:46.137449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:46.137493 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:46.138007 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:46.138012 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:46.139642 (+ 1630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 04:43:46.139804 10799 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:46.135423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:54187 (local address 127.1.226.254:38133)
0928 04:43:46.135467 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:46.135471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:46.135517 (+ 46us) server_negotiation.cc:407] Connection header received
0928 04:43:46.135544 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:46.135545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:46.135559 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:46.135623 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:46.135842 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:46.136430 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:46.137165 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:46.137357 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:46.137421 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:46.139739 (+ 2318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.195:54187: BlockingRecv error: recv got EOF from 127.1.226.195:54187 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 04:43:46.139892 11062 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 04:43:46 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:46 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:46.872093 11066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:46.866652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.193:35245)
0928 04:43:46.866819 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:46.866837 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:46.866896 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:46.867124 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:46.867126 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:46.867296 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:46.867345 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:46.867346 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:46.868146 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:46.868147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:46.868794 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:46.868797 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:46.869220 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:46.869221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:46.869271 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:46.869956 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:46.869962 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:46.872024 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":103,"threads_started":1}
W20260928 04:43:46.872265 11067 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:46.866784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:35245 (local address 127.1.226.254:38133)
0928 04:43:46.866945 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:46.866947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:46.866956 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:43:46.866982 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:46.866983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:46.866993 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:46.867080 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:46.867448 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:46.868062 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:46.868898 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:46.869115 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:46.869185 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:46.872154 (+ 2969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.193:35245: BlockingRecv error: recv got EOF from 127.1.226.193:35245 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
Sep 28 04:43:47 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:47 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:47.014431 11068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:47.009039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.194:59307)
0928 04:43:47.009174 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:47.009195 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:47.009257 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:47.009466 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:47.009468 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:47.009656 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:47.009709 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:47.009711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:47.010521 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:47.010523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:47.011093 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:47.011096 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:47.011500 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:47.011501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:47.011554 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:47.012072 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:47.012076 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:47.014358 (+ 2282us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
W20260928 04:43:47.014547 11067 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:47.009147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:59307 (local address 127.1.226.254:38133)
0928 04:43:47.009258 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:47.009261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:47.009276 (+ 15us) server_negotiation.cc:407] Connection header received
0928 04:43:47.009318 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:47.009319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:47.009345 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:47.009427 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:47.009823 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:47.010397 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:47.011195 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:47.011407 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:47.011479 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:47.014466 (+ 2987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.194:59307: BlockingRecv error: recv got EOF from 127.1.226.194:59307 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 04:43:47 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:47 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:47.145614 11069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:47.140365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.195:51899)
0928 04:43:47.140561 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:47.140573 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:47.140640 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:47.140785 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:47.140785 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:47.140947 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:47.140994 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:47.140996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:47.141617 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:47.141618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:47.142200 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:47.142202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:47.142586 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:47.142587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:47.142630 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:47.143168 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:47.143172 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:47.145546 (+ 2374us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":117,"threads_started":1}
W20260928 04:43:47.145797 11067 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:47.140438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:51899 (local address 127.1.226.254:38133)
0928 04:43:47.140513 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:47.140516 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:47.140640 (+ 124us) server_negotiation.cc:407] Connection header received
0928 04:43:47.140696 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:47.140697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:47.140710 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:47.140761 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:47.141057 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:47.141550 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:47.142316 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:47.142512 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:47.142584 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:47.145690 (+ 3106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.195:51899: BlockingRecv error: recv got EOF from 127.1.226.195:51899 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 04:43:47 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:47 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:47.877636 11070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:47.872742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.193:51541)
0928 04:43:47.873021 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:47.873032 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:47.873120 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:47.873306 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:47.873307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:47.873545 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:47.873602 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:47.873605 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:47.874338 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:47.874339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:47.874918 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:47.874920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:47.875281 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:47.875282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:47.875323 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:47.875781 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:47.875785 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:47.877577 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":91,"threads_started":1}
W20260928 04:43:47.877935 11071 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:47.872854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:51541 (local address 127.1.226.254:38133)
0928 04:43:47.873043 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:47.873045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:47.873160 (+ 115us) server_negotiation.cc:407] Connection header received
0928 04:43:47.873190 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:47.873191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:47.873206 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:47.873280 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:47.873744 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:47.874263 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:47.875010 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:47.875210 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:47.875268 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:47.877834 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.193:51541: BlockingRecv error: recv got EOF from 127.1.226.193:51541 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
Sep 28 04:43:48 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:48 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:48.021481 11072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:48.015145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.194:59535)
0928 04:43:48.015426 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:48.015439 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:48.015705 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:48.015944 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:48.015945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:48.016186 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:48.016239 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:48.016242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:48.017074 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:48.017076 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:48.017744 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:48.017746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:48.018238 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:48.018240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:48.018317 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:48.018946 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:48.018952 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:48.021378 (+ 2426us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":115,"threads_started":1}
W20260928 04:43:48.021533 11071 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:48.015297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:59535 (local address 127.1.226.254:38133)
0928 04:43:48.015413 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:48.015416 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:48.015673 (+ 257us) server_negotiation.cc:407] Connection header received
0928 04:43:48.015813 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:48.015814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:48.015835 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:48.015921 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:48.016345 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:48.016955 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:48.017885 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:48.018128 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:48.018237 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:48.021469 (+ 3232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.194:59535: BlockingRecv error: recv got EOF from 127.1.226.194:59535 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 28 04:43:48 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:48 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:48.150897 11073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:48.146166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.195:34633)
0928 04:43:48.146323 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:48.146337 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:48.146412 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:48.146599 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:48.146600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:48.146773 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:48.146825 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:48.146827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:48.147528 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:48.147529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:48.148065 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:48.148067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:48.148446 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:48.148446 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:48.148489 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:48.148939 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:48.148943 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:48.150839 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":63,"threads_started":1}
W20260928 04:43:48.151018 11071 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:48.146398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:34633 (local address 127.1.226.254:38133)
0928 04:43:48.146471 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:48.146473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:48.146481 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:43:48.146506 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:48.146506 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:48.146519 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:48.146572 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:48.146909 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:48.147430 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:48.148176 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:48.148366 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:48.148422 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:48.150938 (+ 2516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.195:34633: BlockingRecv error: recv got EOF from 127.1.226.195:34633 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 04:43:48 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:48 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:48.883117 11074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:48.878266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.193:35417)
0928 04:43:48.878537 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:48.878548 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:48.878612 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:48.878752 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:48.878753 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:48.878914 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:48.878964 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:48.878966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:48.879643 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:48.879644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:48.880309 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:48.880312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:48.880670 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:48.880672 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:48.880718 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:48.881181 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:48.881185 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:48.883045 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":195,"threads_started":1}
W20260928 04:43:48.883301 11075 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:48.878467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:35417 (local address 127.1.226.254:38133)
0928 04:43:48.878609 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:48.878611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:48.878620 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:43:48.878661 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:48.878662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:48.878672 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:48.878736 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:48.879039 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:48.879542 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:48.880400 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:48.880592 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:48.880647 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:48.883186 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.193:35417: BlockingRecv error: recv got EOF from 127.1.226.193:35417 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
Sep 28 04:43:49 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:49 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:49.026901 11076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:49.022031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.194:49657)
0928 04:43:49.022199 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:49.022208 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:49.022263 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:49.022402 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:49.022403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:49.022560 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:49.022614 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:49.022615 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:49.023280 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:49.023281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:49.023858 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:49.023860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:49.024248 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:49.024249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:49.024295 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:49.024814 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:49.024828 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:49.026840 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":78,"threads_started":1}
W20260928 04:43:49.027043 11075 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:49.022100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:49657 (local address 127.1.226.254:38133)
0928 04:43:49.022202 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:49.022204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:49.022252 (+ 48us) server_negotiation.cc:407] Connection header received
0928 04:43:49.022325 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:49.022326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:49.022336 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:49.022383 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:49.022689 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:49.023203 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:49.023945 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:49.024142 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:49.024234 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:49.026960 (+ 2726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.194:49657: BlockingRecv error: recv got EOF from 127.1.226.194:49657 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 04:43:49 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:49 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:49.156394 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:49.151439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.195:56975)
0928 04:43:49.151608 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:49.151621 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:49.151730 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:49.151887 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:49.151888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:49.152053 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:49.152103 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:49.152104 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:49.152796 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:49.152797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:49.153325 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:49.153327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:49.153658 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:49.153659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:49.153711 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:49.154272 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:49.154285 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:49.156333 (+ 2048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":110,"threads_started":1}
W20260928 04:43:49.156574 11075 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:49.151558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:56975 (local address 127.1.226.254:38133)
0928 04:43:49.151655 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:49.151657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:49.151672 (+ 15us) server_negotiation.cc:407] Connection header received
0928 04:43:49.151797 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:49.151797 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:49.151809 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:49.151880 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:49.152213 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:49.152720 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:49.153403 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:49.153588 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:49.153643 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:49.156475 (+ 2832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.195:56975: BlockingRecv error: recv got EOF from 127.1.226.195:56975 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 04:43:49 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:49 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:49.892794 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:49.883901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.193:45057)
0928 04:43:49.884113 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:49.884139 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:49.884305 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:49.884520 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:49.884521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:49.884719 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:49.884771 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:49.884772 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:49.885513 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:49.885514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:49.886109 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:49.886112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:49.886556 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:49.886557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:49.886609 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:49.890554 (+ 3945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:49.890560 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:49.892734 (+ 2174us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":137,"threads_started":1}
W20260928 04:43:49.893000 11079 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:49.884033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:45057 (local address 127.1.226.254:38133)
0928 04:43:49.884207 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:49.884210 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:49.884255 (+ 45us) server_negotiation.cc:407] Connection header received
0928 04:43:49.884404 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:49.884405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:49.884425 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:49.884495 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:49.884859 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:49.885404 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:49.886221 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:49.886447 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:49.886537 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:49.892921 (+ 6384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.193:45057: BlockingRecv error: recv got EOF from 127.1.226.193:45057 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":105,"threads_started":1}
Sep 28 04:43:50 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:50 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:50.032461 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:50.027462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.194:52309)
0928 04:43:50.027628 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:50.027686 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:50.027759 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:50.027908 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:50.027909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:50.028067 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:50.028115 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:50.028117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:50.028885 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:50.028886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:50.029431 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:50.029433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:50.029863 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:50.029864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:50.029907 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:50.030370 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:50.030375 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:50.032359 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":79,"threads_started":1}
W20260928 04:43:50.032517 11079 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:50.027573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:52309 (local address 127.1.226.254:38133)
0928 04:43:50.027642 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:50.027644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:50.027757 (+ 113us) server_negotiation.cc:407] Connection header received
0928 04:43:50.027819 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:50.027820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:50.027834 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:50.027890 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:50.028225 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:50.028790 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:50.029524 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:50.029781 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:50.029849 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:50.032450 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.194:52309: BlockingRecv error: recv got EOF from 127.1.226.194:52309 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 04:43:50 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:50 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:50.161748 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:50.156918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.195:40345)
0928 04:43:50.157069 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:50.157080 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:50.157168 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:50.157333 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:50.157334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:50.157517 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:50.157567 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:50.157569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:50.158216 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:50.158217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:50.158776 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:50.158778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:50.159113 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:50.159114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:50.159156 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:50.159618 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:50.159623 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:50.161687 (+ 2064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
W20260928 04:43:50.161870 11079 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:50.157019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:40345 (local address 127.1.226.254:38133)
0928 04:43:50.157086 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:50.157088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:50.157169 (+ 81us) server_negotiation.cc:407] Connection header received
0928 04:43:50.157214 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:50.157214 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:50.157226 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:50.157304 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:50.157643 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:50.158135 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:50.158860 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:50.159043 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:50.159097 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:50.161764 (+ 2667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.195:40345: BlockingRecv error: recv got EOF from 127.1.226.195:40345 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 28 04:43:50 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:50 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:50.898370 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:50.893300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.193:57609)
0928 04:43:50.893452 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:50.893468 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:50.893520 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:50.893733 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:50.893734 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:50.893893 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:50.893944 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:50.893945 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:50.894711 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:50.894712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:50.895373 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:50.895375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:50.895841 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:50.895842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:50.895896 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:50.896464 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:50.896470 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:50.898270 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
W20260928 04:43:50.898480 11083 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:50.893436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:57609 (local address 127.1.226.254:38133)
0928 04:43:50.893572 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:50.893574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:50.893583 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:43:50.893613 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:50.893614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:50.893626 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:50.893714 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:50.894044 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:50.894618 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:50.895461 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:50.895750 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:50.895839 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:50.898350 (+ 2511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.193:57609: BlockingRecv error: recv got EOF from 127.1.226.193:57609 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
Sep 28 04:43:51 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:51 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:51.038193 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:51.032956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.194:42239)
0928 04:43:51.033107 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:51.033122 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:51.033216 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:51.033406 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:51.033407 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:51.033581 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:51.033634 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:51.033635 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:51.034423 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:51.034424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:51.035008 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:51.035010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:51.035595 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:51.035596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:51.035661 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:51.036188 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:51.036192 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:51.038106 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":57,"threads_started":1}
W20260928 04:43:51.038372 11083 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:51.033160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:42239 (local address 127.1.226.254:38133)
0928 04:43:51.033280 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:51.033282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:51.033290 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:43:51.033317 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:51.033318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:51.033329 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:51.033384 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:51.033742 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:51.034340 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:51.035162 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:51.035491 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:51.035567 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:51.038261 (+ 2694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.194:42239: BlockingRecv error: recv got EOF from 127.1.226.194:42239 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 28 04:43:51 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:51 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:51.167261 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:51.162237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.195:44319)
0928 04:43:51.162451 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:51.162464 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:51.162564 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:51.162795 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:51.162796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:51.163012 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:51.163058 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:51.163060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:51.163741 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:51.163742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:51.164345 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:51.164347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:51.164729 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:51.164730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:51.164786 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:51.165298 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:51.165303 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:51.167152 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":98,"threads_started":1}
W20260928 04:43:51.167308 11083 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:51.162353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:44319 (local address 127.1.226.254:38133)
0928 04:43:51.162440 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:51.162443 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:51.162541 (+ 98us) server_negotiation.cc:407] Connection header received
0928 04:43:51.162642 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:51.162643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:51.162659 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:51.162789 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:51.163149 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:51.163650 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:51.164435 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:51.164636 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:51.164729 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:51.167241 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.195:44319: BlockingRecv error: recv got EOF from 127.1.226.195:44319 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 04:43:51 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:51 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 04:43:51.904687 11087 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:51.899083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:52673 (local address 127.1.226.254:38133)
0928 04:43:51.899251 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:51.899253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:51.899320 (+ 67us) server_negotiation.cc:407] Connection header received
0928 04:43:51.899391 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:51.899392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:51.899403 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:51.899470 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:51.899838 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:51.900372 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:51.901474 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:51.901723 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:51.901783 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:51.904617 (+ 2834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.193:52673: BlockingRecv error: recv got EOF from 127.1.226.193:52673 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":92,"threads_started":1}
I20260928 04:43:51.904732 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:51.898994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.193:52673)
0928 04:43:51.899242 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:51.899263 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:51.899339 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:51.899493 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:51.899494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:51.899695 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:51.899748 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:51.899750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:51.900487 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:51.900489 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:51.901385 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:51.901389 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:51.901801 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:51.901802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:51.901850 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:51.902387 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:51.902392 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:51.904534 (+ 2142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":182,"threads_started":1}
Sep 28 04:43:52 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:52 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:52.043218 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:52.038776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.194:58023)
0928 04:43:52.038914 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:52.038926 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:52.038995 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:52.039135 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:52.039136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:52.039304 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:52.039352 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:52.039354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:52.040007 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:52.040008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:52.040569 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:52.040570 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:52.040884 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:52.040885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:52.040924 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:52.041366 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:52.041370 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:52.043149 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
W20260928 04:43:52.043291 11087 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:52.038867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:58023 (local address 127.1.226.254:38133)
0928 04:43:52.038936 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:52.038938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:52.038982 (+ 44us) server_negotiation.cc:407] Connection header received
0928 04:43:52.039061 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:52.039062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:52.039074 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:52.039119 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:52.039417 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:52.039934 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:52.040640 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:52.040820 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:52.040870 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:52.043244 (+ 2374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.194:58023: BlockingRecv error: recv got EOF from 127.1.226.194:58023 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 04:43:52 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:52 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:52.172210 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:52.167739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.195:51459)
0928 04:43:52.167887 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:52.167903 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:52.167957 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:52.168086 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:52.168086 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:52.168292 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:52.168339 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:52.168341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:52.169009 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:52.169009 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:52.169539 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:52.169540 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:52.169872 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:52.169873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:52.169910 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:52.170411 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:52.170416 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:52.172098 (+ 1682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
W20260928 04:43:52.172333 11087 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:52.167812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:51459 (local address 127.1.226.254:38133)
0928 04:43:52.167861 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:52.167863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:52.167960 (+ 97us) server_negotiation.cc:407] Connection header received
0928 04:43:52.168009 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:52.168010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:52.168022 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:52.168064 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:52.168433 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:52.168938 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:52.169616 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:52.169810 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:52.169864 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:52.172255 (+ 2391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.195:51459: BlockingRecv error: recv got EOF from 127.1.226.195:51459 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 04:43:52 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:52 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:52.910494 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:52.905281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.193:51241)
0928 04:43:52.905445 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:52.905456 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:52.905541 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:52.905685 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:52.905686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:52.905859 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:52.905911 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:52.905913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:52.906577 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:52.906578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:52.907232 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:52.907234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:52.907724 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:52.907725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:52.907769 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:52.908345 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:52.908350 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:52.910423 (+ 2073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
W20260928 04:43:52.910650 11091 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:52.905376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:51241 (local address 127.1.226.254:38133)
0928 04:43:52.905480 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:52.905482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:52.905511 (+ 29us) server_negotiation.cc:407] Connection header received
0928 04:43:52.905590 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:52.905591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:52.905602 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:52.905661 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:52.905985 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:52.906499 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:52.907319 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:52.907643 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:52.907730 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:52.910562 (+ 2832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.193:51241: BlockingRecv error: recv got EOF from 127.1.226.193:51241 (error 108)
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":49,"threads_started":1}
Sep 28 04:43:53 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:53 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:53.049167 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:53.043920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.194:47901)
0928 04:43:53.044115 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:53.044127 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:53.044235 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:53.044380 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:53.044381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:53.044541 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:53.044589 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:53.044590 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:53.045488 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:53.045489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:53.046201 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:53.046204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:53.046562 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:53.046563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:53.046604 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:53.047089 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:53.047093 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:53.049100 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":79,"threads_started":1}
W20260928 04:43:53.049301 11091 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:53.044043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:47901 (local address 127.1.226.254:38133)
0928 04:43:53.044119 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:53.044121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:53.044232 (+ 111us) server_negotiation.cc:407] Connection header received
0928 04:43:53.044299 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:53.044300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:53.044312 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:53.044355 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:53.044731 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:53.045386 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:53.046283 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:53.046491 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:53.046551 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:53.049213 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.194:47901: BlockingRecv error: recv got EOF from 127.1.226.194:47901 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 04:43:53 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:53 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:53.177809 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:53.172847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.195:46959)
0928 04:43:53.173022 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:53.173045 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:53.173105 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:53.173252 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:53.173253 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:53.173430 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:53.173479 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:53.173480 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:53.174192 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:53.174193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:53.174705 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:53.174706 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:53.175046 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:53.175047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:53.175084 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:53.175574 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:53.175579 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:53.177750 (+ 2171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":116,"threads_started":1}
W20260928 04:43:53.177999 11091 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:53.172935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:46959 (local address 127.1.226.254:38133)
0928 04:43:53.173026 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:53.173028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:53.173104 (+ 76us) server_negotiation.cc:407] Connection header received
0928 04:43:53.173153 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:53.173154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:53.173167 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:53.173232 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:53.173544 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:53.174087 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:53.174792 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:53.174978 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:53.175034 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:53.177933 (+ 2899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.195:46959: BlockingRecv error: recv got EOF from 127.1.226.195:46959 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 04:43:53 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:53 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:53.916029 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:53.910997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.193:59675)
0928 04:43:53.911263 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:53.911276 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:53.911340 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:53.911498 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:53.911499 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:53.911676 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:53.911730 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:53.911732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:53.912468 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:53.912469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:53.913019 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:53.913020 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:53.913388 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:53.913389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:53.913434 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:53.914007 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:53.914011 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:53.915931 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":180,"threads_started":1}
W20260928 04:43:53.916102 11095 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:53.911243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:59675 (local address 127.1.226.254:38133)
0928 04:43:53.911372 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:53.911374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:53.911382 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:43:53.911402 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:53.911403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:53.911414 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:53.911476 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:53.911803 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:53.912382 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:53.913109 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:53.913314 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:53.913370 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:53.916017 (+ 2647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.193:59675: BlockingRecv error: recv got EOF from 127.1.226.193:59675 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":59,"threads_started":1}
Sep 28 04:43:54 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:54 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:54.054325 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:54.049699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.194:39477)
0928 04:43:54.049818 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:54.049830 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:54.049901 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:54.050014 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:54.050015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:54.050167 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:54.050235 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:54.050237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:54.051000 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:54.051000 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:54.051523 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:54.051524 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:54.051837 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:54.051837 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:54.051875 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:54.052354 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:54.052359 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:54.054213 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
W20260928 04:43:54.054370 11095 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:54.049781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:39477 (local address 127.1.226.254:38133)
0928 04:43:54.049825 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:54.049827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:54.049907 (+ 80us) server_negotiation.cc:407] Connection header received
0928 04:43:54.049947 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:54.049948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:54.049959 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:54.050003 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:54.050293 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:54.050926 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:54.051592 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:54.051774 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:54.051823 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:54.054306 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.194:39477: BlockingRecv error: recv got EOF from 127.1.226.194:39477 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 04:43:54 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:54 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:54.183843 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:54.178353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.195:60803)
0928 04:43:54.178558 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:54.178647 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:54.178764 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:54.178961 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:54.178962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:54.179186 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:54.179241 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:54.179244 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:54.180070 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:54.180071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:54.180765 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:54.180768 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:54.181326 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:54.181327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:54.181407 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:54.182015 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:54.182025 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:54.183781 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":105,"threads_started":1}
W20260928 04:43:54.184013 11095 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:54.178468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:60803 (local address 127.1.226.254:38133)
0928 04:43:54.178579 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:54.178581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:54.178740 (+ 159us) server_negotiation.cc:407] Connection header received
0928 04:43:54.178854 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:54.178855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:54.178875 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:54.178953 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:54.179355 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:54.179933 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:54.180896 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:54.181177 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:54.181270 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:54.183930 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.195:60803: BlockingRecv error: recv got EOF from 127.1.226.195:60803 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 28 04:43:54 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:54 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:54.926160 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:54.916519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.193:60473)
0928 04:43:54.916692 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:54.916704 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:54.916767 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:54.916908 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:54.916909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:54.917071 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:54.917123 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:54.917124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:54.917836 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:54.917838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:54.918409 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:54.918412 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:54.918807 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:54.918808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:54.918854 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:54.923749 (+ 4895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:54.923755 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:54.926063 (+ 2308us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":97,"threads_started":1}
W20260928 04:43:54.926283 11099 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:54.916616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:60473 (local address 127.1.226.254:38133)
0928 04:43:54.916743 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:54.916745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:54.916755 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:43:54.916828 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:54.916829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:54.916839 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:54.916889 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:54.917205 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:54.917739 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:54.918500 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:54.918729 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:54.918786 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:54.926198 (+ 7412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.193:60473: BlockingRecv error: recv got EOF from 127.1.226.193:60473 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
Sep 28 04:43:55 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:55 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:55.060458 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:55.054914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.194:42665)
0928 04:43:55.055094 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:55.055107 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:55.055189 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:55.055364 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:55.055365 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:55.055628 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:55.055682 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:55.055685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:55.056535 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:55.056537 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:55.057112 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:55.057115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:55.057533 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:55.057534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:55.057603 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:55.058152 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:55.058158 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:55.060388 (+ 2230us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":94,"threads_started":1}
W20260928 04:43:55.060622 11099 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:55.054975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:42665 (local address 127.1.226.254:38133)
0928 04:43:55.055054 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:55.055056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:55.055182 (+ 126us) server_negotiation.cc:407] Connection header received
0928 04:43:55.055267 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:55.055268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:55.055282 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:55.055358 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:55.055781 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:55.056425 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:55.057236 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:55.057443 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:55.057533 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:55.060521 (+ 2988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.194:42665: BlockingRecv error: recv got EOF from 127.1.226.194:42665 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 04:43:55 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:55 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:55.189713 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:55.184476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.195:45547)
0928 04:43:55.184663 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:55.184676 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:55.184741 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:55.184928 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:55.184930 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:55.185096 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:55.185142 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:55.185143 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:55.185830 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:55.185831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:55.186469 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:55.186471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:55.186931 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:55.186932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:55.186993 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:55.187448 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:55.187453 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:55.189586 (+ 2133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":125,"threads_started":1}
W20260928 04:43:55.189725 11099 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:55.184606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:45547 (local address 127.1.226.254:38133)
0928 04:43:55.184662 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:55.184664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:55.184727 (+ 63us) server_negotiation.cc:407] Connection header received
0928 04:43:55.184793 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:55.184794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:55.184807 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:55.184874 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:55.185229 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:55.185738 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:55.186574 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:55.186837 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:55.186930 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:55.189667 (+ 2737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.195:45547: BlockingRecv error: recv got EOF from 127.1.226.195:45547 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 04:43:55 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:55 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:55.931555 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:55.926714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.193:58499)
0928 04:43:55.926863 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:55.926877 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:55.926952 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:55.927106 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:55.927107 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:55.927274 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:55.927352 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:55.927355 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:55.928035 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:55.928036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:55.928610 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:55.928612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:55.929002 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:55.929002 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:55.929045 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:55.929608 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:55.929613 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:55.931497 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":66,"threads_started":1}
W20260928 04:43:55.931789 11103 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:55.926767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:58499 (local address 127.1.226.254:38133)
0928 04:43:55.926899 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:55.926902 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:55.926933 (+ 31us) server_negotiation.cc:407] Connection header received
0928 04:43:55.927009 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:55.927010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:55.927022 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:55.927099 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:55.927446 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:55.927959 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:55.928731 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:55.928930 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:55.928988 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:55.931665 (+ 2677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.193:58499: BlockingRecv error: recv got EOF from 127.1.226.193:58499 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
Sep 28 04:43:56 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:56 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:56.065742 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:56.060954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.194:47843)
0928 04:43:56.061146 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:56.061155 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:56.061200 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:56.061369 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:56.061370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:56.061530 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:56.061577 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:56.061579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:56.062296 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:56.062297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:56.062862 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:56.062864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:56.063230 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:56.063230 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:56.063272 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:56.063773 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:56.063777 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:56.065660 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":114,"threads_started":1}
W20260928 04:43:56.065874 11103 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:56.061136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:47843 (local address 127.1.226.254:38133)
0928 04:43:56.061231 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:56.061232 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:56.061242 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:43:56.061271 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:56.061271 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:56.061285 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:56.061364 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:56.061703 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:56.062213 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:56.062957 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:56.063155 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:56.063215 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:56.065797 (+ 2582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.194:47843: BlockingRecv error: recv got EOF from 127.1.226.194:47843 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 04:43:56 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:56 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:56.195708 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:56.190237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.195:37673)
0928 04:43:56.190380 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:56.190395 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:56.190470 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:56.190662 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:56.190663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:56.190893 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:56.190954 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:56.190957 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:56.191783 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:56.191784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:56.192385 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:56.192387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:56.192848 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:56.192849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:56.192905 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:56.193406 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:56.193411 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:56.195619 (+ 2208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
W20260928 04:43:56.195819 11103 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:56.190443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:37673 (local address 127.1.226.254:38133)
0928 04:43:56.190515 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:56.190517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:56.190530 (+ 13us) server_negotiation.cc:407] Connection header received
0928 04:43:56.190566 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:56.190567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:56.190581 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:56.190642 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:56.191079 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:56.191709 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:56.192548 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:56.192744 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:56.192810 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:56.195744 (+ 2934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.195:37673: BlockingRecv error: recv got EOF from 127.1.226.195:37673 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 04:43:56 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:56 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:56.937636 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:56.932106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.193:56725)
0928 04:43:56.932327 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:56.932347 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:56.932404 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:56.932574 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:56.932574 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:56.932792 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:56.932841 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:56.932843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:56.933607 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:56.933608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:56.934263 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:56.934267 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:56.934705 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:56.934706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:56.934763 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:56.935380 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:56.935389 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:56.937567 (+ 2178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":139,"threads_started":1}
W20260928 04:43:56.937821 11107 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:56.932224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:56725 (local address 127.1.226.254:38133)
0928 04:43:56.932386 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:56.932388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:56.932398 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:43:56.932478 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:56.932479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:56.932492 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:56.932560 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:56.932947 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:56.933519 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:56.934354 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:56.934552 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:56.934615 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:56.937689 (+ 3074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.193:56725: BlockingRecv error: recv got EOF from 127.1.226.193:56725 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":77,"threads_started":1}
Sep 28 04:43:57 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:57 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:57.071274 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:57.066311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.194:35909)
0928 04:43:57.066492 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:57.066503 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:57.066564 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:57.066704 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:57.066704 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:57.066875 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:57.066920 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:57.066922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:57.067572 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:57.067573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:57.068122 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:57.068124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:57.068534 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:57.068534 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:57.068581 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:57.069107 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:57.069118 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:57.071176 (+ 2058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":108,"threads_started":1}
W20260928 04:43:57.071331 11107 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:57.066373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:35909 (local address 127.1.226.254:38133)
0928 04:43:57.066441 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:57.066443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:57.066567 (+ 124us) server_negotiation.cc:407] Connection header received
0928 04:43:57.066611 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:57.066612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:57.066624 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:57.066681 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:57.067000 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:57.067498 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:57.068257 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:57.068466 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:57.068520 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:57.071283 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.194:35909: BlockingRecv error: recv got EOF from 127.1.226.194:35909 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 04:43:57 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:57 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:57.201930 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:57.196226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.195:57623)
0928 04:43:57.196368 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:57.196381 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:57.196440 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:57.196645 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:57.196646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:57.196867 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:57.196928 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:57.196930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:57.197631 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:57.197632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:57.198621 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:57.198624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:57.199100 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:57.199101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:57.199170 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:57.199727 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:57.199733 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:57.201856 (+ 2123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":70,"threads_started":1}
W20260928 04:43:57.202062 11107 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:57.196448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:57623 (local address 127.1.226.254:38133)
0928 04:43:57.196503 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:57.196505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:57.196512 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:43:57.196536 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:57.196537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:57.196552 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:57.196611 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:57.197028 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:57.197553 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:57.198747 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:57.199010 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:57.199082 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:57.201979 (+ 2897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.195:57623: BlockingRecv error: recv got EOF from 127.1.226.195:57623 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 04:43:57 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:57 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:57.943043 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:57.938154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.193:36115)
0928 04:43:57.938332 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:57.938342 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:57.938393 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:57.938590 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:57.938591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:57.938789 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:57.938841 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:57.938842 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:57.939584 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:57.939585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:57.940210 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:57.940213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:57.940564 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:57.940565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:57.940611 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:57.941153 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:57.941159 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:57.942972 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":82,"threads_started":1}
W20260928 04:43:57.943161 11111 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:57.938243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:36115 (local address 127.1.226.254:38133)
0928 04:43:57.938380 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:57.938382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:57.938391 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:43:57.938441 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:57.938442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:57.938453 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:57.938527 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:57.938972 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:57.939482 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:57.940294 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:57.940489 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:57.940543 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:57.943082 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.193:36115: BlockingRecv error: recv got EOF from 127.1.226.193:36115 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
Sep 28 04:43:58 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:58 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:58.076810 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:58.071828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.194:53061)
0928 04:43:58.071981 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:58.071998 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:58.072054 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:58.072214 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:58.072214 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:58.072392 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:58.072448 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:58.072450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:58.073156 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:58.073157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:58.073730 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:58.073732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:58.074152 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:58.074154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:58.074204 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:58.074655 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:58.074659 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:58.076743 (+ 2084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
W20260928 04:43:58.076926 11111 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:58.071953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:53061 (local address 127.1.226.254:38133)
0928 04:43:58.072009 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:58.072010 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:58.072040 (+ 30us) server_negotiation.cc:407] Connection header received
0928 04:43:58.072113 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:58.072113 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:58.072123 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:58.072208 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:58.072528 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:58.073063 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:58.073834 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:58.074045 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:58.074124 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:58.076822 (+ 2698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.194:53061: BlockingRecv error: recv got EOF from 127.1.226.194:53061 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 04:43:58 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:58 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:58.207577 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:58.202491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.195:32787)
0928 04:43:58.202692 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:58.202702 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:58.202757 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:58.202964 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:58.202965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:58.203172 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:58.203224 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:58.203226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:58.204006 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:58.204007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:58.204616 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:58.204618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:58.205029 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:58.205030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:58.205078 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:58.205617 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:58.205621 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:58.207517 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":96,"threads_started":1}
W20260928 04:43:58.207680 11111 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:58.202622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:32787 (local address 127.1.226.254:38133)
0928 04:43:58.202701 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:58.202703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:58.202839 (+ 136us) server_negotiation.cc:407] Connection header received
0928 04:43:58.202878 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:58.202879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:58.202893 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:58.202941 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:58.203300 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:58.203933 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:58.204700 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:58.204913 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:58.205026 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:58.207627 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.195:32787: BlockingRecv error: recv got EOF from 127.1.226.195:32787 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 04:43:58 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:58 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:58.948732 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:58.943605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.193:36731)
0928 04:43:58.943798 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:58.943813 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:58.943885 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:58.944097 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:58.944099 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:58.944296 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:58.944345 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:58.944347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:58.945073 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:58.945074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:58.945631 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:58.945634 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:58.946006 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:58.946007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:58.946055 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:58.946551 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:58.946555 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:58.948660 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":123,"threads_started":1}
W20260928 04:43:58.948860 11115 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:58.943802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:36731 (local address 127.1.226.254:38133)
0928 04:43:58.943940 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:58.943942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:58.943949 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:43:58.943969 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:58.943970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:58.943981 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:58.944052 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:58.944425 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:58.944972 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:58.945738 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:58.945932 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:58.945986 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:58.948776 (+ 2790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.193:36731: BlockingRecv error: recv got EOF from 127.1.226.193:36731 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
Sep 28 04:43:59 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:59 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:59.082180 11116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:59.077376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.194:41479)
0928 04:43:59.077547 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:59.077558 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:59.077618 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:59.077791 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:59.077792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:59.077970 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:59.078020 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:59.078022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:59.078679 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:59.078680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:59.079232 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:59.079234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:59.079588 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:59.079590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:59.079683 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:59.080142 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:59.080146 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:59.082090 (+ 1944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":97,"threads_started":1}
W20260928 04:43:59.082332 11115 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:59.077534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:41479 (local address 127.1.226.254:38133)
0928 04:43:59.077603 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:59.077605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:59.077613 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:43:59.077687 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:59.077688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:59.077704 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:59.077767 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:59.078094 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:59.078601 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:59.079312 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:59.079494 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:59.079550 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:59.082239 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.194:41479: BlockingRecv error: recv got EOF from 127.1.226.194:41479 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 04:43:59 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:59 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:59.213539 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:59.208295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.195:41605)
0928 04:43:59.208463 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:59.208477 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:59.208586 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:59.208835 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:59.208836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:59.209026 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:59.209073 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:59.209075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:59.209781 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:59.209782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:59.210349 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:59.210351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:59.210725 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:59.210726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:59.210795 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:59.211298 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:59.211302 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:59.213437 (+ 2135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":103,"threads_started":1}
W20260928 04:43:59.213586 11115 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:59.208411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:41605 (local address 127.1.226.254:38133)
0928 04:43:59.208467 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:59.208469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:59.208564 (+ 95us) server_negotiation.cc:407] Connection header received
0928 04:43:59.208663 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:59.208664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:59.208680 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:59.208820 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:59.209158 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:59.209685 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:59.210442 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:59.210635 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:59.210709 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:59.213521 (+ 2812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.195:41605: BlockingRecv error: recv got EOF from 127.1.226.195:41605 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 04:43:59 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:43:59 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:43:59.962870 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:43:59.949462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.193:54081)
0928 04:43:59.949738 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 04:43:59.949759 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 04:43:59.949832 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:43:59.950017 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:43:59.950018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:43:59.950205 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:43:59.950258 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:59.950260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:59.951289 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:59.951291 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:59.951843 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:43:59.951845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:43:59.952293 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:43:59.952294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:43:59.952356 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:59.960421 (+ 8065us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:43:59.960428 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:43:59.962804 (+ 2376us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":99,"threads_started":1}
W20260928 04:43:59.963018 11119 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:43:59.949604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:54081 (local address 127.1.226.254:38133)
0928 04:43:59.949768 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 04:43:59.949771 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:43:59.949814 (+ 43us) server_negotiation.cc:407] Connection header received
0928 04:43:59.949896 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:43:59.949896 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:43:59.949909 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:43:59.949962 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:43:59.950370 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:59.951181 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:59.951952 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:43:59.952211 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:43:59.952293 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:43:59.962923 (+ 10630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.193:54081: BlockingRecv error: recv got EOF from 127.1.226.193:54081 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
Sep 28 04:44:00 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:44:00 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:44:00.088232 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:00.082855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.194:54881)
0928 04:44:00.083020 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 04:44:00.083031 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:44:00.083102 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:44:00.083271 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:44:00.083272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:44:00.083441 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:44:00.083487 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:00.083489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:00.084499 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:00.084500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:00.085176 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:00.085178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:00.085572 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:00.085573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:00.085625 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:00.086141 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:44:00.086146 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:44:00.088100 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":97,"threads_started":1}
W20260928 04:44:00.088351 11119 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:44:00.082957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:54881 (local address 127.1.226.254:38133)
0928 04:44:00.083035 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:00.083037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:00.083082 (+ 45us) server_negotiation.cc:407] Connection header received
0928 04:44:00.083172 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:00.083173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:00.083183 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:00.083247 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:44:00.083602 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:00.084386 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:00.085274 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:00.085468 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:00.085532 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:00.088267 (+ 2735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.194:54881: BlockingRecv error: recv got EOF from 127.1.226.194:54881 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 04:44:00 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:44:00 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.195@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:44:00.219475 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:00.214122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.195:42487)
0928 04:44:00.214342 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 04:44:00.214353 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:44:00.214435 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:44:00.214578 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:44:00.214579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:44:00.214759 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:44:00.214803 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:00.214805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:00.215480 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:00.215480 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:00.216037 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:00.216039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:00.216498 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:00.216499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:00.216553 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:00.217091 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:44:00.217096 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:44:00.219405 (+ 2309us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":80,"threads_started":1}
W20260928 04:44:00.219638 11119 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:44:00.214277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:42487 (local address 127.1.226.254:38133)
0928 04:44:00.214352 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:00.214354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:00.214433 (+ 79us) server_negotiation.cc:407] Connection header received
0928 04:44:00.214486 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:00.214487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:00.214500 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:00.214554 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:44:00.214890 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:00.215403 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:00.216119 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:00.216395 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:00.216463 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:00.219532 (+ 3069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.195:42487: BlockingRecv error: recv got EOF from 127.1.226.195:42487 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 04:44:00 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:44:00 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.193@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:44:00.968930 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:00.963580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.193:53407)
0928 04:44:00.963766 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 04:44:00.963782 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 04:44:00.963862 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:44:00.964055 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:44:00.964056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:44:00.964324 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:44:00.964404 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:00.964407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:00.965392 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:00.965393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:00.965954 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:00.965956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:00.966342 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:00.966342 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:00.966391 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:00.966830 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:44:00.966835 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:44:00.968859 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":87,"threads_started":1}
W20260928 04:44:00.969070 11123 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:44:00.963756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:53407 (local address 127.1.226.254:38133)
0928 04:44:00.963909 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:00.963911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:00.963921 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:44:00.963950 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:00.963950 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:00.963964 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:00.964043 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:44:00.964540 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:00.965313 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:00.966057 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:00.966271 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:00.966333 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:00.968959 (+ 2626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.193:53407: BlockingRecv error: recv got EOF from 127.1.226.193:53407 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
Sep 28 04:44:01 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:44:01 dist-test-slave-j5c3 krb5kdc[10065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.226.194@KRBTEST.COM for oryx/127.1.226.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:44:01.093688 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:01.088771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38133 (local address 127.1.226.194:45489)
0928 04:44:01.088942 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 04:44:01.088955 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:44:01.089020 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:44:01.089164 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:44:01.089165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:44:01.089470 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:44:01.089523 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:01.089525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:01.090198 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:01.090199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:01.090776 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:01.090778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:01.091154 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:01.091155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:01.091200 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:01.091643 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:44:01.091647 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:44:01.093613 (+ 1966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.226.254:38133: Server oryx/127.1.226.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":122,"threads_started":1}
W20260928 04:44:01.093798 11123 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:44:01.088886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:45489 (local address 127.1.226.254:38133)
0928 04:44:01.088932 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:01.088934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:01.089022 (+ 88us) server_negotiation.cc:407] Connection header received
0928 04:44:01.089076 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:01.089076 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:01.089087 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:01.089144 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:44:01.089597 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:01.090111 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:01.090876 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:01.091080 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:01.091173 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:01.093725 (+ 2552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.226.194:45489: BlockingRecv error: recv got EOF from 127.1.226.194:45489 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:44:01.129892 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 10648
I20260928 04:44:01.134593 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 10800
I20260928 04:44:01.138760 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 10933
I20260928 04:44:01.143287 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 10578
2026-09-28T04:44:01Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17565 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T04:44:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:44:01Z Disabled control of system clock
I20260928 04:44:01.171515 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:46311
--webserver_interface=127.1.226.254
--webserver_port=0
--builtin_ntp_servers=127.1.226.212:36589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:46311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 04:44:01.255784 11128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:44:01.256029 11128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:44:01.256062 11128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:44:01.257295 11128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:44:01.257367 11128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:44:01.257395 11128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:44:01.257421 11128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:44:01.257452 11128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:44:01.258806 11128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:36589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:46311
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:46311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:44:01.259078 11128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:44:01.259379 11128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:44:01.265061 11136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:44:01.265126 11133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:44:01.265151 11134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:44:01.265237 11128 server_base.cc:1034] running on GCE node
I20260928 04:44:01.265580 11128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:44:01.265879 11128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:44:01.267022 11128 hybrid_clock.cc:648] HybridClock initialized: now 1790570641267009 us; error 30 us; skew 500 ppm
I20260928 04:44:01.268431 11128 webserver.cc:460] Webserver started at http://127.1.226.254:40721/ using document root <none> and password file <none>
I20260928 04:44:01.268693 11128 fs_manager.cc:359] Metadata directory not provided
I20260928 04:44:01.268743 11128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:44:01.268873 11128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:44:01.270001 11128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "bb8af9ff58ea410db0f51dd89a369b54"
format_stamp: "Formatted at 2026-09-28 04:44:01 on dist-test-slave-j5c3"
server_key: "12e85e731a3cf2a6731c65e4b738281b"
server_key_iv: "f2f50336d61b898654e53d493fdf0930"
server_key_version: "encryptionkey@0"
I20260928 04:44:01.270370 11128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "bb8af9ff58ea410db0f51dd89a369b54"
format_stamp: "Formatted at 2026-09-28 04:44:01 on dist-test-slave-j5c3"
server_key: "12e85e731a3cf2a6731c65e4b738281b"
server_key_iv: "f2f50336d61b898654e53d493fdf0930"
server_key_version: "encryptionkey@0"
I20260928 04:44:01.271687 11128 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:44:01.272482 11142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:01.272675 11128 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:44:01.272766 11128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "bb8af9ff58ea410db0f51dd89a369b54"
format_stamp: "Formatted at 2026-09-28 04:44:01 on dist-test-slave-j5c3"
server_key: "12e85e731a3cf2a6731c65e4b738281b"
server_key_iv: "f2f50336d61b898654e53d493fdf0930"
server_key_version: "encryptionkey@0"
I20260928 04:44:01.272825 11128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:44:01.285485 11128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:44:01.285770 11128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:44:01.285873 11128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:44:01.289911 11128 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:46311
I20260928 04:44:01.289985 11194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:46311 every 8 connection(s)
I20260928 04:44:01.290324 11128 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:44:01.291066 11195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:44:01.293773 11195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb8af9ff58ea410db0f51dd89a369b54: Bootstrap starting.
I20260928 04:44:01.294358 11195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb8af9ff58ea410db0f51dd89a369b54: Neither blocks nor log segments found. Creating new log.
I20260928 04:44:01.294600 11195 log.cc:824] T 00000000000000000000000000000000 P bb8af9ff58ea410db0f51dd89a369b54: Log is configured to *not* fsync() on all Append() calls
I20260928 04:44:01.295239 11195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb8af9ff58ea410db0f51dd89a369b54: No bootstrap required, opened a new log
I20260928 04:44:01.295982 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 11128
I20260928 04:44:01.296067 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:44:01.296284 1931 external_mini_cluster.cc:1428] Setting key 38c274593016d88c59364fce9d120231
I20260928 04:44:01.297439 11195 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb8af9ff58ea410db0f51dd89a369b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8af9ff58ea410db0f51dd89a369b54" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 46311 } }
I20260928 04:44:01.297614 11195 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb8af9ff58ea410db0f51dd89a369b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:44:01.297652 11195 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb8af9ff58ea410db0f51dd89a369b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb8af9ff58ea410db0f51dd89a369b54, State: Initialized, Role: FOLLOWER
I20260928 04:44:01.297816 11195 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb8af9ff58ea410db0f51dd89a369b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8af9ff58ea410db0f51dd89a369b54" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 46311 } }
I20260928 04:44:01.297883 11195 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb8af9ff58ea410db0f51dd89a369b54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:44:01.297961 11195 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb8af9ff58ea410db0f51dd89a369b54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:44:01.298020 11195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb8af9ff58ea410db0f51dd89a369b54 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:44:01.298789 11195 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb8af9ff58ea410db0f51dd89a369b54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8af9ff58ea410db0f51dd89a369b54" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 46311 } }
I20260928 04:44:01.298897 11195 leader_election.cc:302] T 00000000000000000000000000000000 P bb8af9ff58ea410db0f51dd89a369b54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb8af9ff58ea410db0f51dd89a369b54; no voters:
I20260928 04:44:01.299057 11195 leader_election.cc:288] T 00000000000000000000000000000000 P bb8af9ff58ea410db0f51dd89a369b54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:44:01.299126 11200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb8af9ff58ea410db0f51dd89a369b54 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:44:01.299301 11200 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb8af9ff58ea410db0f51dd89a369b54 [term 1 LEADER]: Becoming Leader. State: Replica: bb8af9ff58ea410db0f51dd89a369b54, State: Running, Role: LEADER
I20260928 04:44:01.299305 11195 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb8af9ff58ea410db0f51dd89a369b54 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:44:01.299465 11200 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb8af9ff58ea410db0f51dd89a369b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8af9ff58ea410db0f51dd89a369b54" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 46311 } }
I20260928 04:44:01.300077 11201 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb8af9ff58ea410db0f51dd89a369b54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb8af9ff58ea410db0f51dd89a369b54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8af9ff58ea410db0f51dd89a369b54" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 46311 } } }
I20260928 04:44:01.300200 11201 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb8af9ff58ea410db0f51dd89a369b54 [sys.catalog]: This master's current role is: LEADER
I20260928 04:44:01.300598 11202 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb8af9ff58ea410db0f51dd89a369b54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb8af9ff58ea410db0f51dd89a369b54. Latest consensus state: current_term: 1 leader_uuid: "bb8af9ff58ea410db0f51dd89a369b54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb8af9ff58ea410db0f51dd89a369b54" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 46311 } } }
I20260928 04:44:01.300863 11202 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb8af9ff58ea410db0f51dd89a369b54 [sys.catalog]: This master's current role is: LEADER
I20260928 04:44:01.300794 11209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:44:01.301366 11209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:44:01.303162 11209 catalog_manager.cc:1347] Generated new cluster ID: e5e3bdcc910a4b47ab0f8d36a24e3fb9
I20260928 04:44:01.303210 11209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:44:01.303619 11199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:01.296901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36500 (local address 127.1.226.254:46311)
0928 04:44:01.297169 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:01.297179 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:01.297219 (+ 40us) server_negotiation.cc:407] Connection header received
0928 04:44:01.297392 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:01.297400 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:01.297487 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:01.297639 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:44:01.298130 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:01.299891 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:01.301057 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:01.301721 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:01.301845 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:01.302758 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:44:01.302761 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:44:01.302762 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:44:01.302968 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:44:01.303010 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:44:01.303025 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:44:01.303282 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:01.303327 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 04:44:01.303404 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":103,"threads_started":1}
I20260928 04:44:01.313339 11209 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:44:01.313890 11209 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:44:01.316763 11209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb8af9ff58ea410db0f51dd89a369b54: Generated new TSK 0
I20260928 04:44:01.316962 11209 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:44:01.319980 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:0
--local_ip_for_outbound_sockets=127.1.226.193
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:46311
--builtin_ntp_servers=127.1.226.212:36589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 04:44:01.402619 11219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:44:01.402859 11219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:44:01.402899 11219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:44:01.404067 11219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:44:01.404129 11219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:44:01.404196 11219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:44:01.405623 11219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:36589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:46311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:44:01.405884 11219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:44:01.406186 11219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:44:01.412130 11219 server_base.cc:1034] running on GCE node
W20260928 04:44:01.412114 11227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:44:01.412137 11224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:44:01.412127 11225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:44:01.414979 11219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:44:01.415306 11219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:44:01.416505 11219 hybrid_clock.cc:648] HybridClock initialized: now 1790570641416483 us; error 41 us; skew 500 ppm
I20260928 04:44:01.417888 11219 webserver.cc:460] Webserver started at http://127.1.226.193:39373/ using document root <none> and password file <none>
I20260928 04:44:01.418134 11219 fs_manager.cc:359] Metadata directory not provided
I20260928 04:44:01.418180 11219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:44:01.418339 11219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:44:01.419425 11219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/data/instance:
uuid: "efaf79dcedf44695951d6f4ea123e598"
format_stamp: "Formatted at 2026-09-28 04:44:01 on dist-test-slave-j5c3"
server_key: "be5b56da10377ab0bb94c1d45e266e96"
server_key_iv: "5964cd66c95afd7025d78c33d290d59f"
server_key_version: "encryptionkey@0"
I20260928 04:44:01.419787 11219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance:
uuid: "efaf79dcedf44695951d6f4ea123e598"
format_stamp: "Formatted at 2026-09-28 04:44:01 on dist-test-slave-j5c3"
server_key: "be5b56da10377ab0bb94c1d45e266e96"
server_key_iv: "5964cd66c95afd7025d78c33d290d59f"
server_key_version: "encryptionkey@0"
I20260928 04:44:01.421191 11219 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:44:01.421972 11233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:01.422123 11219 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:44:01.422228 11219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "efaf79dcedf44695951d6f4ea123e598"
format_stamp: "Formatted at 2026-09-28 04:44:01 on dist-test-slave-j5c3"
server_key: "be5b56da10377ab0bb94c1d45e266e96"
server_key_iv: "5964cd66c95afd7025d78c33d290d59f"
server_key_version: "encryptionkey@0"
I20260928 04:44:01.422305 11219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:44:01.436026 11219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:44:01.436381 11219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:44:01.436540 11219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:44:01.436812 11219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:44:01.437181 11219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:44:01.437227 11219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:01.437268 11219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:44:01.437321 11219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:01.443382 11219 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:37441
I20260928 04:44:01.443425 11346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:37441 every 8 connection(s)
I20260928 04:44:01.443785 11219 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
I20260928 04:44:01.445209 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 11219
I20260928 04:44:01.445305 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance
I20260928 04:44:01.445495 1931 external_mini_cluster.cc:1428] Setting key 94717cf03a1d509a91beebfe740c44bc
I20260928 04:44:01.446519 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:0
--local_ip_for_outbound_sockets=127.1.226.194
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:46311
--builtin_ntp_servers=127.1.226.212:36589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 04:44:01.450100 11199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:01.444600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:56337 (local address 127.1.226.254:46311)
0928 04:44:01.444672 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:01.444673 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:01.444909 (+ 236us) server_negotiation.cc:407] Connection header received
0928 04:44:01.445727 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:01.445729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:01.445745 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:01.445801 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:44:01.446705 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:01.447267 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:01.448532 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:01.448790 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:01.448859 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:01.449828 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:44:01.449829 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:44:01.449829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:44:01.449834 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:44:01.449847 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:44:01.449851 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:44:01.449907 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:01.450011 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 04:44:01.450038 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 04:44:01.450420 11349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:01.444595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:46311 (local address 127.1.226.193:56337)
0928 04:44:01.444787 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 04:44:01.444820 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 04:44:01.445582 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:44:01.445898 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:44:01.445902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:44:01.445918 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:44:01.446565 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:01.446573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:01.447361 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:01.447362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:01.448435 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:01.448438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:01.448901 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:01.448903 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:01.449066 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:01.449659 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:44:01.449659 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:44:01.449701 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:44:01.449729 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:44:01.449928 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:44:01.449934 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:44:01.449937 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:44:01.450005 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 04:44:01.450143 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
I20260928 04:44:01.450937 11347 heartbeater.cc:347] Connected to a master server at 127.1.226.254:46311
I20260928 04:44:01.451033 11347 heartbeater.cc:464] Registering TS with master...
I20260928 04:44:01.451259 11347 heartbeater.cc:511] Master 127.1.226.254:46311 requested a full tablet report, sending...
I20260928 04:44:01.451805 11159 ts_manager.cc:194] Registered new tserver with Master: efaf79dcedf44695951d6f4ea123e598 (127.1.226.193:37441)
I20260928 04:44:01.452479 11159 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.226.193:56337
W20260928 04:44:01.530839 11350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:44:01.531100 11350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:44:01.531136 11350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:44:01.532363 11350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:44:01.532421 11350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:44:01.532459 11350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:44:01.533793 11350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:36589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:46311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:44:01.534055 11350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:44:01.534360 11350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:44:01.539474 11358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:44:01.539474 11355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:44:01.539547 11356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:44:01.539716 11350 server_base.cc:1034] running on GCE node
I20260928 04:44:01.539956 11350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:44:01.540264 11350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:44:01.541450 11350 hybrid_clock.cc:648] HybridClock initialized: now 1790570641541441 us; error 31 us; skew 500 ppm
I20260928 04:44:01.542976 11350 webserver.cc:460] Webserver started at http://127.1.226.194:37277/ using document root <none> and password file <none>
I20260928 04:44:01.543244 11350 fs_manager.cc:359] Metadata directory not provided
I20260928 04:44:01.543294 11350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:44:01.543465 11350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:44:01.544632 11350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/data/instance:
uuid: "7f5230aa54d94970860ea4e9f10bc8a9"
format_stamp: "Formatted at 2026-09-28 04:44:01 on dist-test-slave-j5c3"
server_key: "3aee631eda8b09872e2e4de9ee1b21ac"
server_key_iv: "ad422e271579922289ee768032a9440b"
server_key_version: "encryptionkey@0"
I20260928 04:44:01.544992 11350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance:
uuid: "7f5230aa54d94970860ea4e9f10bc8a9"
format_stamp: "Formatted at 2026-09-28 04:44:01 on dist-test-slave-j5c3"
server_key: "3aee631eda8b09872e2e4de9ee1b21ac"
server_key_iv: "ad422e271579922289ee768032a9440b"
server_key_version: "encryptionkey@0"
I20260928 04:44:01.546357 11350 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 04:44:01.547312 11364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:01.547508 11350 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:44:01.547585 11350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "7f5230aa54d94970860ea4e9f10bc8a9"
format_stamp: "Formatted at 2026-09-28 04:44:01 on dist-test-slave-j5c3"
server_key: "3aee631eda8b09872e2e4de9ee1b21ac"
server_key_iv: "ad422e271579922289ee768032a9440b"
server_key_version: "encryptionkey@0"
I20260928 04:44:01.547633 11350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:44:01.557148 11350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:44:01.557406 11350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:44:01.557508 11350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:44:01.557695 11350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:44:01.557976 11350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:44:01.558005 11350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:01.558027 11350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:44:01.558040 11350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:01.563944 11350 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:37951
I20260928 04:44:01.563985 11477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:37951 every 8 connection(s)
I20260928 04:44:01.564388 11350 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
I20260928 04:44:01.570243 11199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:01.565237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:54909 (local address 127.1.226.254:46311)
0928 04:44:01.565336 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:01.565339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:01.565546 (+ 207us) server_negotiation.cc:407] Connection header received
0928 04:44:01.566066 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:01.566068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:01.566087 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:01.566161 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:44:01.566928 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:01.567494 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:01.568757 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:01.568968 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:01.569032 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:01.569965 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:44:01.569966 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:44:01.569966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:44:01.569973 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:44:01.569987 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:44:01.569991 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:44:01.570049 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:01.570153 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 04:44:01.570180 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 04:44:01.570444 11480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:01.565218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:46311 (local address 127.1.226.194:54909)
0928 04:44:01.565405 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 04:44:01.565448 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 04:44:01.565951 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:44:01.566234 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:44:01.566237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:44:01.566245 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:44:01.566812 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:01.566817 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:01.567586 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:01.567586 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:01.568545 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:01.568548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:01.569055 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:01.569056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:01.569169 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:01.569790 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:44:01.569790 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:44:01.569825 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:44:01.569851 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:44:01.570062 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:44:01.570067 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:44:01.570070 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:44:01.570132 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 04:44:01.570229 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260928 04:44:01.571084 11478 heartbeater.cc:347] Connected to a master server at 127.1.226.254:46311
I20260928 04:44:01.571188 11478 heartbeater.cc:464] Registering TS with master...
I20260928 04:44:01.571388 11478 heartbeater.cc:511] Master 127.1.226.254:46311 requested a full tablet report, sending...
I20260928 04:44:01.571874 11159 ts_manager.cc:194] Registered new tserver with Master: 7f5230aa54d94970860ea4e9f10bc8a9 (127.1.226.194:37951)
I20260928 04:44:01.572317 11159 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.226.194:54909
I20260928 04:44:01.572551 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 11350
I20260928 04:44:01.572665 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance
I20260928 04:44:01.572860 1931 external_mini_cluster.cc:1428] Setting key 10c44934f0a123ad040467c3c4310b86
I20260928 04:44:01.573807 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:0
--local_ip_for_outbound_sockets=127.1.226.195
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:46311
--builtin_ntp_servers=127.1.226.212:36589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 04:44:01.654597 11481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:44:01.654800 11481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:44:01.654821 11481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:44:01.655973 11481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:44:01.656015 11481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:44:01.656040 11481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:44:01.657385 11481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:36589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:46311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:44:01.657634 11481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:44:01.657929 11481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:44:01.662911 11489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:44:01.662947 11486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:44:01.662958 11481 server_base.cc:1034] running on GCE node
W20260928 04:44:01.662983 11487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:44:01.663311 11481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:44:01.663563 11481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:44:01.664732 11481 hybrid_clock.cc:648] HybridClock initialized: now 1790570641664710 us; error 41 us; skew 500 ppm
I20260928 04:44:01.666057 11481 webserver.cc:460] Webserver started at http://127.1.226.195:33509/ using document root <none> and password file <none>
I20260928 04:44:01.666488 11481 fs_manager.cc:359] Metadata directory not provided
I20260928 04:44:01.666581 11481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:44:01.666813 11481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:44:01.667948 11481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/data/instance:
uuid: "69367ec3b9e44537a19be1406a301200"
format_stamp: "Formatted at 2026-09-28 04:44:01 on dist-test-slave-j5c3"
server_key: "6205e83f1402e0da150664659c0517ce"
server_key_iv: "7d1155202466207ad1360df71ac9db84"
server_key_version: "encryptionkey@0"
I20260928 04:44:01.668340 11481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance:
uuid: "69367ec3b9e44537a19be1406a301200"
format_stamp: "Formatted at 2026-09-28 04:44:01 on dist-test-slave-j5c3"
server_key: "6205e83f1402e0da150664659c0517ce"
server_key_iv: "7d1155202466207ad1360df71ac9db84"
server_key_version: "encryptionkey@0"
I20260928 04:44:01.669807 11481 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:44:01.670604 11495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:01.670815 11481 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:44:01.670922 11481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "69367ec3b9e44537a19be1406a301200"
format_stamp: "Formatted at 2026-09-28 04:44:01 on dist-test-slave-j5c3"
server_key: "6205e83f1402e0da150664659c0517ce"
server_key_iv: "7d1155202466207ad1360df71ac9db84"
server_key_version: "encryptionkey@0"
I20260928 04:44:01.671010 11481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:44:01.685554 11481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:44:01.685889 11481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:44:01.686045 11481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:44:01.686269 11481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:44:01.686619 11481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:44:01.686678 11481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:01.686720 11481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:44:01.686748 11481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:01.692396 11481 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:43321
I20260928 04:44:01.692507 11608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:43321 every 8 connection(s)
I20260928 04:44:01.692823 11481 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
I20260928 04:44:01.698076 11199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:01.693594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:56767 (local address 127.1.226.254:46311)
0928 04:44:01.693678 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:01.693682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:01.693891 (+ 209us) server_negotiation.cc:407] Connection header received
0928 04:44:01.694392 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:01.694393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:01.694405 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:01.694483 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:44:01.695168 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:01.695725 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:01.696796 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:01.696990 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:01.697052 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:01.697830 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:44:01.697832 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:44:01.697832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:44:01.697837 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:44:01.697852 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:44:01.697856 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:44:01.697902 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:01.697989 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 04:44:01.698014 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 04:44:01.698298 11611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:01.693543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:46311 (local address 127.1.226.195:56767)
0928 04:44:01.693745 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 04:44:01.693790 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 04:44:01.694286 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:44:01.694555 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:44:01.694558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:44:01.694567 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:44:01.695044 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:01.695048 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:01.695835 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:01.695836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:01.696716 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:01.696719 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:01.697069 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:01.697070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:01.697178 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:01.697690 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:44:01.697691 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:44:01.697719 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:44:01.697743 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:44:01.697917 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:44:01.697922 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:44:01.697924 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:44:01.697977 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 04:44:01.698062 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
I20260928 04:44:01.698823 11609 heartbeater.cc:347] Connected to a master server at 127.1.226.254:46311
I20260928 04:44:01.698956 11609 heartbeater.cc:464] Registering TS with master...
I20260928 04:44:01.699012 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 11481
I20260928 04:44:01.699105 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance
I20260928 04:44:01.699167 11609 heartbeater.cc:511] Master 127.1.226.254:46311 requested a full tablet report, sending...
I20260928 04:44:01.699342 1931 external_mini_cluster.cc:1428] Setting key 482fc2153e28caf03f2c4e4fb62f3de4
I20260928 04:44:01.699558 11159 ts_manager.cc:194] Registered new tserver with Master: 69367ec3b9e44537a19be1406a301200 (127.1.226.195:43321)
I20260928 04:44:01.700047 11159 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.226.195:56767
I20260928 04:44:01.700273 1931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 04:44:01.702168 11199 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:44:01.701717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36510 (local address 127.1.226.254:46311)
0928 04:44:01.701756 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:01.701758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:01.701872 (+ 114us) server_negotiation.cc:407] Connection header received
0928 04:44:01.701931 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:01.701932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:01.701942 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:01.701995 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:44:01.702090 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36510: BlockingRecv error: recv got EOF from 127.0.0.1:36510 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 04:44:01.702265 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 11219
I20260928 04:44:01.707311 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 11350
I20260928 04:44:01.711647 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 11481
I20260928 04:44:01.715862 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 11128
2026-09-28T04:44:01Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (571 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T04:44:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:44:01Z Disabled control of system clock
I20260928 04:44:01.742983 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:33309
--webserver_interface=127.1.226.254
--webserver_port=0
--builtin_ntp_servers=127.1.226.212:43203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:33309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 04:44:01.826328 11620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:44:01.826550 11620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:44:01.826574 11620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:44:01.827855 11620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:44:01.827907 11620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:44:01.827922 11620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:44:01.827939 11620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:44:01.827952 11620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:44:01.829306 11620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:43203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:33309
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:33309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:44:01.829535 11620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:44:01.829804 11620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:44:01.834985 11626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:44:01.835216 11620 server_base.cc:1034] running on GCE node
W20260928 04:44:01.834985 11625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:44:01.835033 11628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:44:01.835589 11620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:44:01.835903 11620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:44:01.837095 11620 hybrid_clock.cc:648] HybridClock initialized: now 1790570641837078 us; error 35 us; skew 500 ppm
I20260928 04:44:01.838452 11620 webserver.cc:460] Webserver started at http://127.1.226.254:33591/ using document root <none> and password file <none>
I20260928 04:44:01.838739 11620 fs_manager.cc:359] Metadata directory not provided
I20260928 04:44:01.838815 11620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:44:01.838981 11620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:44:01.840090 11620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "62738ae555e34f7aac0666c64f77ed58"
format_stamp: "Formatted at 2026-09-28 04:44:01 on dist-test-slave-j5c3"
server_key: "84cce6a429f380fb5ca8b0a970c04530"
server_key_iv: "57ab0afe2607d58196ff0d4fec793c50"
server_key_version: "encryptionkey@0"
I20260928 04:44:01.840528 11620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "62738ae555e34f7aac0666c64f77ed58"
format_stamp: "Formatted at 2026-09-28 04:44:01 on dist-test-slave-j5c3"
server_key: "84cce6a429f380fb5ca8b0a970c04530"
server_key_iv: "57ab0afe2607d58196ff0d4fec793c50"
server_key_version: "encryptionkey@0"
I20260928 04:44:01.842044 11620 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:44:01.842859 11634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:01.843091 11620 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:44:01.843223 11620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "62738ae555e34f7aac0666c64f77ed58"
format_stamp: "Formatted at 2026-09-28 04:44:01 on dist-test-slave-j5c3"
server_key: "84cce6a429f380fb5ca8b0a970c04530"
server_key_iv: "57ab0afe2607d58196ff0d4fec793c50"
server_key_version: "encryptionkey@0"
I20260928 04:44:01.843317 11620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:44:01.846925 11620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:44:01.847195 11620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:44:01.847339 11620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:44:01.851265 11620 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:33309
I20260928 04:44:01.851297 11686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:33309 every 8 connection(s)
I20260928 04:44:01.851648 11620 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:44:01.852380 11687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:44:01.855093 11687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62738ae555e34f7aac0666c64f77ed58: Bootstrap starting.
I20260928 04:44:01.855638 11687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62738ae555e34f7aac0666c64f77ed58: Neither blocks nor log segments found. Creating new log.
I20260928 04:44:01.855873 11687 log.cc:824] T 00000000000000000000000000000000 P 62738ae555e34f7aac0666c64f77ed58: Log is configured to *not* fsync() on all Append() calls
I20260928 04:44:01.856654 11687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62738ae555e34f7aac0666c64f77ed58: No bootstrap required, opened a new log
I20260928 04:44:01.857411 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 11620
I20260928 04:44:01.857523 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:44:01.857743 1931 external_mini_cluster.cc:1428] Setting key aee6cc8e03d9aad176829a835aea6f1a
I20260928 04:44:01.859035 11687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62738ae555e34f7aac0666c64f77ed58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62738ae555e34f7aac0666c64f77ed58" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33309 } }
I20260928 04:44:01.859189 11687 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62738ae555e34f7aac0666c64f77ed58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:44:01.859215 11687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62738ae555e34f7aac0666c64f77ed58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62738ae555e34f7aac0666c64f77ed58, State: Initialized, Role: FOLLOWER
I20260928 04:44:01.859426 11687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62738ae555e34f7aac0666c64f77ed58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62738ae555e34f7aac0666c64f77ed58" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33309 } }
I20260928 04:44:01.859529 11687 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62738ae555e34f7aac0666c64f77ed58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:44:01.859571 11687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62738ae555e34f7aac0666c64f77ed58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:44:01.859620 11687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62738ae555e34f7aac0666c64f77ed58 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:44:01.860555 11687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62738ae555e34f7aac0666c64f77ed58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62738ae555e34f7aac0666c64f77ed58" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33309 } }
I20260928 04:44:01.860682 11687 leader_election.cc:302] T 00000000000000000000000000000000 P 62738ae555e34f7aac0666c64f77ed58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62738ae555e34f7aac0666c64f77ed58; no voters:
I20260928 04:44:01.860862 11687 leader_election.cc:288] T 00000000000000000000000000000000 P 62738ae555e34f7aac0666c64f77ed58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:44:01.860865 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:01.858370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60262 (local address 127.1.226.254:33309)
0928 04:44:01.858575 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:01.858583 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:01.858666 (+ 83us) server_negotiation.cc:407] Connection header received
0928 04:44:01.859042 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:01.859050 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:01.859101 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:01.859295 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:44:01.860086 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:44:01.860090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:44:01.860090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:44:01.860094 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:44:01.860133 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:44:01.860143 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:44:01.860338 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:01.860452 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 04:44:01.860549 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260928 04:44:01.861184 11687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62738ae555e34f7aac0666c64f77ed58 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:44:01.861518 11692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62738ae555e34f7aac0666c64f77ed58 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:44:01.861665 11692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62738ae555e34f7aac0666c64f77ed58 [term 1 LEADER]: Becoming Leader. State: Replica: 62738ae555e34f7aac0666c64f77ed58, State: Running, Role: LEADER
I20260928 04:44:01.861773 11692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62738ae555e34f7aac0666c64f77ed58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62738ae555e34f7aac0666c64f77ed58" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33309 } }
I20260928 04:44:01.862145 11692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62738ae555e34f7aac0666c64f77ed58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62738ae555e34f7aac0666c64f77ed58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62738ae555e34f7aac0666c64f77ed58" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33309 } } }
I20260928 04:44:01.862237 11692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62738ae555e34f7aac0666c64f77ed58 [sys.catalog]: This master's current role is: LEADER
I20260928 04:44:01.862432 11692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62738ae555e34f7aac0666c64f77ed58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62738ae555e34f7aac0666c64f77ed58. Latest consensus state: current_term: 1 leader_uuid: "62738ae555e34f7aac0666c64f77ed58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62738ae555e34f7aac0666c64f77ed58" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33309 } } }
I20260928 04:44:01.862507 11692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62738ae555e34f7aac0666c64f77ed58 [sys.catalog]: This master's current role is: LEADER
W20260928 04:44:01.863111 11707 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 62738ae555e34f7aac0666c64f77ed58: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:44:01.863161 11707 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:44:01.863240 11704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:44:01.863428 11704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:44:01.865067 11704 catalog_manager.cc:1347] Generated new cluster ID: b38aa7a9189e43b0862f446eaea83732
I20260928 04:44:01.865131 11704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:44:01.882426 11704 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:44:01.882642 11704 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:44:01.885699 11704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62738ae555e34f7aac0666c64f77ed58: Generated new TSK 0
I20260928 04:44:01.885931 11704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:44:01.891045 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:0
--local_ip_for_outbound_sockets=127.1.226.193
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:33309
--builtin_ntp_servers=127.1.226.212:43203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 04:44:01.972712 11711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:44:01.972949 11711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:44:01.972987 11711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:44:01.974182 11711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:44:01.974246 11711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:44:01.974290 11711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:44:01.975718 11711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:43203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:33309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:44:01.975968 11711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:44:01.976307 11711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:44:01.982314 11717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:44:01.982376 11719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:44:01.982285 11716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:44:01.982745 11711 server_base.cc:1034] running on GCE node
I20260928 04:44:01.983009 11711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:44:01.983312 11711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:44:01.984493 11711 hybrid_clock.cc:648] HybridClock initialized: now 1790570641984471 us; error 37 us; skew 500 ppm
I20260928 04:44:01.986030 11711 webserver.cc:460] Webserver started at http://127.1.226.193:33947/ using document root <none> and password file <none>
I20260928 04:44:01.986301 11711 fs_manager.cc:359] Metadata directory not provided
I20260928 04:44:01.986393 11711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:44:01.986552 11711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:44:01.987730 11711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/data/instance:
uuid: "1b36dd0c842242df86da1713067ed1f4"
format_stamp: "Formatted at 2026-09-28 04:44:01 on dist-test-slave-j5c3"
server_key: "e0afe069dbb00325f31e82c7e9a4f8b8"
server_key_iv: "a2f774f83f770bc34fd1083747bdb714"
server_key_version: "encryptionkey@0"
I20260928 04:44:01.988131 11711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance:
uuid: "1b36dd0c842242df86da1713067ed1f4"
format_stamp: "Formatted at 2026-09-28 04:44:01 on dist-test-slave-j5c3"
server_key: "e0afe069dbb00325f31e82c7e9a4f8b8"
server_key_iv: "a2f774f83f770bc34fd1083747bdb714"
server_key_version: "encryptionkey@0"
I20260928 04:44:01.989737 11711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:44:01.990592 11725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:01.990803 11711 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:44:01.990919 11711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "1b36dd0c842242df86da1713067ed1f4"
format_stamp: "Formatted at 2026-09-28 04:44:01 on dist-test-slave-j5c3"
server_key: "e0afe069dbb00325f31e82c7e9a4f8b8"
server_key_iv: "a2f774f83f770bc34fd1083747bdb714"
server_key_version: "encryptionkey@0"
I20260928 04:44:01.991011 11711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:44:01.994724 11711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:44:01.995033 11711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:44:01.995193 11711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:44:01.995426 11711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:44:01.995836 11711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:44:01.995895 11711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:01.995941 11711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:44:01.995970 11711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:02.002764 11711 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:43905
I20260928 04:44:02.002872 11838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:43905 every 8 connection(s)
I20260928 04:44:02.003239 11711 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
I20260928 04:44:02.006016 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 11711
I20260928 04:44:02.006160 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance
I20260928 04:44:02.006407 1931 external_mini_cluster.cc:1428] Setting key ca85ca43f19a290fd934a8edc38ed292
I20260928 04:44:02.007015 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:02.004032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:43173 (local address 127.1.226.254:33309)
0928 04:44:02.004109 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:02.004111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:02.004432 (+ 321us) server_negotiation.cc:407] Connection header received
0928 04:44:02.004999 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:02.005000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:02.005011 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:02.005070 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:44:02.006592 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:44:02.006594 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:44:02.006594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:44:02.006600 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:44:02.006622 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:44:02.006627 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:44:02.006714 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:02.006911 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 04:44:02.006932 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 04:44:02.007166 11841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:02.004045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:33309 (local address 127.1.226.193:43173)
0928 04:44:02.004295 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 04:44:02.004335 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 04:44:02.004877 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:44:02.005152 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:44:02.005157 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:44:02.005170 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:44:02.006359 (+ 1189us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:44:02.006360 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:44:02.006406 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:44:02.006448 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:44:02.006731 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:44:02.006740 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:44:02.006744 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:44:02.006842 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 04:44:02.006916 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":129,"threads_started":1}
I20260928 04:44:02.007685 11839 heartbeater.cc:347] Connected to a master server at 127.1.226.254:33309
I20260928 04:44:02.007787 11839 heartbeater.cc:464] Registering TS with master...
I20260928 04:44:02.007805 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:0
--local_ip_for_outbound_sockets=127.1.226.194
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:33309
--builtin_ntp_servers=127.1.226.212:43203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 04:44:02.007963 11839 heartbeater.cc:511] Master 127.1.226.254:33309 requested a full tablet report, sending...
I20260928 04:44:02.008654 11651 ts_manager.cc:194] Registered new tserver with Master: 1b36dd0c842242df86da1713067ed1f4 (127.1.226.193:43905)
W20260928 04:44:02.091727 11842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:44:02.091998 11842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:44:02.092037 11842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:44:02.093515 11842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:44:02.093575 11842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:44:02.093629 11842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:44:02.095186 11842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:43203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:33309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:44:02.095391 11842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:44:02.095656 11842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:44:02.100778 11847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:44:02.100822 11850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:44:02.100781 11848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:44:02.100975 11842 server_base.cc:1034] running on GCE node
I20260928 04:44:02.101307 11842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:44:02.101612 11842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:44:02.102780 11842 hybrid_clock.cc:648] HybridClock initialized: now 1790570642102752 us; error 37 us; skew 500 ppm
I20260928 04:44:02.104275 11842 webserver.cc:460] Webserver started at http://127.1.226.194:44239/ using document root <none> and password file <none>
I20260928 04:44:02.104561 11842 fs_manager.cc:359] Metadata directory not provided
I20260928 04:44:02.104686 11842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:44:02.104892 11842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:44:02.106150 11842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/data/instance:
uuid: "ee3dab7d0eea4855b8d1f939e079a35d"
format_stamp: "Formatted at 2026-09-28 04:44:02 on dist-test-slave-j5c3"
server_key: "bbf5fc81b02ad8c86d2de94d5fcfea5e"
server_key_iv: "dcd86e7b3b589bf5adbbbd3b58f72a2e"
server_key_version: "encryptionkey@0"
I20260928 04:44:02.106576 11842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance:
uuid: "ee3dab7d0eea4855b8d1f939e079a35d"
format_stamp: "Formatted at 2026-09-28 04:44:02 on dist-test-slave-j5c3"
server_key: "bbf5fc81b02ad8c86d2de94d5fcfea5e"
server_key_iv: "dcd86e7b3b589bf5adbbbd3b58f72a2e"
server_key_version: "encryptionkey@0"
I20260928 04:44:02.108067 11842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260928 04:44:02.108927 11856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:02.109127 11842 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:44:02.109210 11842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "ee3dab7d0eea4855b8d1f939e079a35d"
format_stamp: "Formatted at 2026-09-28 04:44:02 on dist-test-slave-j5c3"
server_key: "bbf5fc81b02ad8c86d2de94d5fcfea5e"
server_key_iv: "dcd86e7b3b589bf5adbbbd3b58f72a2e"
server_key_version: "encryptionkey@0"
I20260928 04:44:02.109302 11842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:44:02.113027 11842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:44:02.113304 11842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:44:02.113445 11842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:44:02.113656 11842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:44:02.113979 11842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:44:02.114032 11842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:02.114074 11842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:44:02.114099 11842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:02.119843 11842 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:34073
I20260928 04:44:02.119880 11969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:34073 every 8 connection(s)
I20260928 04:44:02.120329 11842 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
I20260928 04:44:02.122849 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 11842
I20260928 04:44:02.122942 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance
I20260928 04:44:02.123170 1931 external_mini_cluster.cc:1428] Setting key 91dfd6ab9a00f2e24707c36775e5c074
I20260928 04:44:02.124006 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:02.121241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:33309 (local address 127.1.226.254:33309)
0928 04:44:02.121312 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:02.121314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:02.121667 (+ 353us) server_negotiation.cc:407] Connection header received
0928 04:44:02.122343 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:02.122344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:02.122350 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:02.122407 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:44:02.123765 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:44:02.123767 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:44:02.123767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:44:02.123773 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:44:02.123791 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:44:02.123796 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:44:02.123848 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:02.123920 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 04:44:02.123935 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 04:44:02.124126 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:0
--local_ip_for_outbound_sockets=127.1.226.195
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:33309
--builtin_ntp_servers=127.1.226.212:43203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 04:44:02.124372 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:02.121223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:33309 (local address 127.1.226.194:33309)
0928 04:44:02.121449 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 04:44:02.121549 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0928 04:44:02.122229 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:44:02.122484 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:44:02.122488 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:44:02.122497 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:44:02.123612 (+ 1115us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:44:02.123613 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:44:02.123642 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:44:02.123670 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:44:02.123844 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:44:02.123848 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:44:02.123851 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:44:02.123929 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 04:44:02.123989 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
I20260928 04:44:02.124913 11970 heartbeater.cc:347] Connected to a master server at 127.1.226.254:33309
I20260928 04:44:02.125013 11970 heartbeater.cc:464] Registering TS with master...
I20260928 04:44:02.125168 11970 heartbeater.cc:511] Master 127.1.226.254:33309 requested a full tablet report, sending...
I20260928 04:44:02.125597 11651 ts_manager.cc:194] Registered new tserver with Master: ee3dab7d0eea4855b8d1f939e079a35d (127.1.226.194:34073)
W20260928 04:44:02.209791 11973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:44:02.210031 11973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:44:02.210067 11973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:44:02.211261 11973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:44:02.211326 11973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:44:02.211375 11973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:44:02.212855 11973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:43203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:33309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:44:02.213158 11973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:44:02.213491 11973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:44:02.218632 11973 server_base.cc:1034] running on GCE node
W20260928 04:44:02.218562 11978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:44:02.218562 11979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:44:02.218565 11981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:44:02.219152 11973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:44:02.219426 11973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:44:02.220595 11973 hybrid_clock.cc:648] HybridClock initialized: now 1790570642220550 us; error 72 us; skew 500 ppm
I20260928 04:44:02.222034 11973 webserver.cc:460] Webserver started at http://127.1.226.195:37513/ using document root <none> and password file <none>
I20260928 04:44:02.222297 11973 fs_manager.cc:359] Metadata directory not provided
I20260928 04:44:02.222369 11973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:44:02.222515 11973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:44:02.223577 11973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/data/instance:
uuid: "8d6c496189c54bc2a67efdb4432fdebf"
format_stamp: "Formatted at 2026-09-28 04:44:02 on dist-test-slave-j5c3"
server_key: "5a562174660b532f27e5ddb4b5d82a2b"
server_key_iv: "6024ef4419acbdb07eb7c9047eb232ed"
server_key_version: "encryptionkey@0"
I20260928 04:44:02.223953 11973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance:
uuid: "8d6c496189c54bc2a67efdb4432fdebf"
format_stamp: "Formatted at 2026-09-28 04:44:02 on dist-test-slave-j5c3"
server_key: "5a562174660b532f27e5ddb4b5d82a2b"
server_key_iv: "6024ef4419acbdb07eb7c9047eb232ed"
server_key_version: "encryptionkey@0"
I20260928 04:44:02.225349 11973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:44:02.226159 11987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:02.226330 11973 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:44:02.226435 11973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "8d6c496189c54bc2a67efdb4432fdebf"
format_stamp: "Formatted at 2026-09-28 04:44:02 on dist-test-slave-j5c3"
server_key: "5a562174660b532f27e5ddb4b5d82a2b"
server_key_iv: "6024ef4419acbdb07eb7c9047eb232ed"
server_key_version: "encryptionkey@0"
I20260928 04:44:02.226522 11973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:44:02.230126 11973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:44:02.230396 11973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:44:02.230533 11973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:44:02.230741 11973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:44:02.231056 11973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:44:02.231109 11973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:02.231154 11973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:44:02.231189 11973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:02.237123 11973 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:44263
I20260928 04:44:02.237161 12100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:44263 every 8 connection(s)
I20260928 04:44:02.237504 11973 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
I20260928 04:44:02.239461 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 11973
I20260928 04:44:02.239565 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance
I20260928 04:44:02.239751 1931 external_mini_cluster.cc:1428] Setting key 707c0b5e4c2179050dcff79e9ff20001
I20260928 04:44:02.240978 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:02.238407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:50683 (local address 127.1.226.254:33309)
0928 04:44:02.238498 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:02.238500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:02.238734 (+ 234us) server_negotiation.cc:407] Connection header received
0928 04:44:02.239209 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:02.239210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:02.239216 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:02.239280 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:44:02.240719 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:44:02.240720 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:44:02.240721 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:44:02.240726 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:44:02.240741 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:44:02.240746 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:44:02.240794 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:02.240890 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 04:44:02.240907 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 04:44:02.241225 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:02.238353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:33309 (local address 127.1.226.195:50683)
0928 04:44:02.238596 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 04:44:02.238631 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 04:44:02.239119 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:44:02.239371 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:44:02.239376 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:44:02.239388 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:44:02.240533 (+ 1145us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:44:02.240534 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:44:02.240572 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:44:02.240602 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:44:02.240803 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:44:02.240808 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:44:02.240811 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:44:02.240901 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 04:44:02.240967 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":121,"threads_started":1}
I20260928 04:44:02.241935 12101 heartbeater.cc:347] Connected to a master server at 127.1.226.254:33309
I20260928 04:44:02.242048 12101 heartbeater.cc:464] Registering TS with master...
I20260928 04:44:02.242195 12101 heartbeater.cc:511] Master 127.1.226.254:33309 requested a full tablet report, sending...
I20260928 04:44:02.242575 11651 ts_manager.cc:194] Registered new tserver with Master: 8d6c496189c54bc2a67efdb4432fdebf (127.1.226.195:44263)
I20260928 04:44:02.251680 1931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 04:44:02.253690 11691 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:44:02.253145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60274 (local address 127.1.226.254:33309)
0928 04:44:02.253217 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:02.253220 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:02.253355 (+ 135us) server_negotiation.cc:407] Connection header received
0928 04:44:02.253400 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:02.253401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:02.253406 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:02.253455 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:44:02.253619 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60274: BlockingRecv error: recv got EOF from 127.0.0.1:60274 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 04:44:02.253808 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 11711
I20260928 04:44:02.258723 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 11842
I20260928 04:44:02.263291 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 11973
I20260928 04:44:02.267499 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 11620
2026-09-28T04:44:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (551 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:44:02 dist-test-slave-j5c3 krb5kdc[12111](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:44:02 dist-test-slave-j5c3 krb5kdc[12111](info): set up 2 sockets
Sep 28 04:44:02 dist-test-slave-j5c3 krb5kdc[12111](info): commencing operation
krb5kdc: starting...
W20260928 04:44:03.361218 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:44:03 dist-test-slave-j5c3 krb5kdc[12111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:44:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:44:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:44:03.488142 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:33561
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:39303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:33561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:44:03.569168 12124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:44:03.569401 12124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:44:03.569425 12124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:44:03.570590 12124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:44:03.570633 12124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:44:03.570647 12124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:44:03.570660 12124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:44:03.570672 12124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:44:03.572144 12124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:39303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:33561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:33561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:44:03.572439 12124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:44:03.572695 12124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:44:03.577732 12132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:44:03.577762 12129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:44:03.577870 12124 server_base.cc:1034] running on GCE node
W20260928 04:44:03.577757 12130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:44:03.578191 12124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:44:03.578552 12124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:44:03.579782 12124 hybrid_clock.cc:648] HybridClock initialized: now 1790570643579755 us; error 41 us; skew 500 ppm
Sep 28 04:44:03 dist-test-slave-j5c3 krb5kdc[12111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570643, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:44:03.582374 12124 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:44:03.582899 12124 webserver.cc:460] Webserver started at http://127.1.226.254:37873/ using document root <none> and password file <none>
I20260928 04:44:03.583128 12124 fs_manager.cc:359] Metadata directory not provided
I20260928 04:44:03.583195 12124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:44:03.583350 12124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:44:03.584477 12124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "a3482117f03e4bb49a45c0e5a97928ee"
format_stamp: "Formatted at 2026-09-28 04:44:03 on dist-test-slave-j5c3"
server_key: "163a11edf2469935453105561f9c7dfc"
server_key_iv: "63b1ff5c0a4001bded96f35e2976291d"
server_key_version: "encryptionkey@0"
I20260928 04:44:03.584841 12124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "a3482117f03e4bb49a45c0e5a97928ee"
format_stamp: "Formatted at 2026-09-28 04:44:03 on dist-test-slave-j5c3"
server_key: "163a11edf2469935453105561f9c7dfc"
server_key_iv: "63b1ff5c0a4001bded96f35e2976291d"
server_key_version: "encryptionkey@0"
I20260928 04:44:03.586138 12124 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:44:03.586905 12139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:03.587059 12124 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:44:03.587167 12124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "a3482117f03e4bb49a45c0e5a97928ee"
format_stamp: "Formatted at 2026-09-28 04:44:03 on dist-test-slave-j5c3"
server_key: "163a11edf2469935453105561f9c7dfc"
server_key_iv: "63b1ff5c0a4001bded96f35e2976291d"
server_key_version: "encryptionkey@0"
I20260928 04:44:03.587242 12124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:44:03.595713 12124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:44:03.597715 12124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:44:03.597880 12124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:44:03.601940 12124 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:33561
I20260928 04:44:03.601977 12191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:33561 every 8 connection(s)
I20260928 04:44:03.602378 12124 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:44:03.603245 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 12124
I20260928 04:44:03.603358 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:44:03.603557 1931 external_mini_cluster.cc:1428] Setting key 3c103bc7d86cb31f6f1b2f7c35b657d6
I20260928 04:44:03.603669 12192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:44:03.606878 12192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3482117f03e4bb49a45c0e5a97928ee: Bootstrap starting.
I20260928 04:44:03.607601 12192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3482117f03e4bb49a45c0e5a97928ee: Neither blocks nor log segments found. Creating new log.
I20260928 04:44:03.607895 12192 log.cc:824] T 00000000000000000000000000000000 P a3482117f03e4bb49a45c0e5a97928ee: Log is configured to *not* fsync() on all Append() calls
I20260928 04:44:03.608628 12192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3482117f03e4bb49a45c0e5a97928ee: No bootstrap required, opened a new log
Sep 28 04:44:03 dist-test-slave-j5c3 krb5kdc[12111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:44:03.610858 12192 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3482117f03e4bb49a45c0e5a97928ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3482117f03e4bb49a45c0e5a97928ee" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33561 } }
I20260928 04:44:03.611020 12192 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3482117f03e4bb49a45c0e5a97928ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:44:03.611093 12192 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3482117f03e4bb49a45c0e5a97928ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3482117f03e4bb49a45c0e5a97928ee, State: Initialized, Role: FOLLOWER
I20260928 04:44:03.611230 12192 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3482117f03e4bb49a45c0e5a97928ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3482117f03e4bb49a45c0e5a97928ee" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33561 } }
I20260928 04:44:03.611306 12192 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3482117f03e4bb49a45c0e5a97928ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:44:03.611328 12192 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3482117f03e4bb49a45c0e5a97928ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:44:03.611392 12192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3482117f03e4bb49a45c0e5a97928ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:44:03.611922 12192 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3482117f03e4bb49a45c0e5a97928ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3482117f03e4bb49a45c0e5a97928ee" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33561 } }
I20260928 04:44:03.612036 12192 leader_election.cc:302] T 00000000000000000000000000000000 P a3482117f03e4bb49a45c0e5a97928ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3482117f03e4bb49a45c0e5a97928ee; no voters:
I20260928 04:44:03.612226 12192 leader_election.cc:288] T 00000000000000000000000000000000 P a3482117f03e4bb49a45c0e5a97928ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:44:03.612421 12198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3482117f03e4bb49a45c0e5a97928ee [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:44:03.612516 12192 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3482117f03e4bb49a45c0e5a97928ee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:44:03.612633 12198 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3482117f03e4bb49a45c0e5a97928ee [term 1 LEADER]: Becoming Leader. State: Replica: a3482117f03e4bb49a45c0e5a97928ee, State: Running, Role: LEADER
I20260928 04:44:03.612824 12198 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3482117f03e4bb49a45c0e5a97928ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3482117f03e4bb49a45c0e5a97928ee" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33561 } }
I20260928 04:44:03.613237 12197 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3482117f03e4bb49a45c0e5a97928ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3482117f03e4bb49a45c0e5a97928ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3482117f03e4bb49a45c0e5a97928ee" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33561 } } }
I20260928 04:44:03.613242 12199 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3482117f03e4bb49a45c0e5a97928ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3482117f03e4bb49a45c0e5a97928ee. Latest consensus state: current_term: 1 leader_uuid: "a3482117f03e4bb49a45c0e5a97928ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3482117f03e4bb49a45c0e5a97928ee" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33561 } } }
I20260928 04:44:03.613438 12199 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3482117f03e4bb49a45c0e5a97928ee [sys.catalog]: This master's current role is: LEADER
I20260928 04:44:03.613428 12197 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3482117f03e4bb49a45c0e5a97928ee [sys.catalog]: This master's current role is: LEADER
I20260928 04:44:03.613817 12207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:44:03.614603 12207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:44:03.615952 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:03.604337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36744 (local address 127.1.226.254:33561)
0928 04:44:03.604504 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:03.604513 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:03.604552 (+ 39us) server_negotiation.cc:407] Connection header received
0928 04:44:03.605143 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:03.605151 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:03.605204 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:03.605326 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:44:03.606106 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:03.607033 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:03.608113 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:03.608648 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:03.608867 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:03.611775 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:44:03.611779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:44:03.611780 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:44:03.611792 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:44:03.614172 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:44:03.614618 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:44:03.614622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:44:03.614622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:44:03.614742 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:44:03.614988 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:44:03.614989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:44:03.614989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:44:03.615296 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:44:03.615445 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:03.615576 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 04:44:03.615724 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260928 04:44:03.616353 12207 catalog_manager.cc:1347] Generated new cluster ID: 6159894feccb4c7aa645457fdf55d684
I20260928 04:44:03.616442 12207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:44:03.638053 12207 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:44:03.638650 12207 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:44:03.648635 12207 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3482117f03e4bb49a45c0e5a97928ee: Generated new TSK 0
I20260928 04:44:03.648883 12207 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.226.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
I20260928 04:44:03.678678 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:0
--local_ip_for_outbound_sockets=127.1.226.193
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:33561
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:39303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:44:03.761023 12218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:44:03.761278 12218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:44:03.761317 12218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:44:03.762490 12218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:44:03.762564 12218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:44:03.762609 12218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:44:03.764114 12218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:39303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:33561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:44:03.764427 12218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:44:03.764704 12218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:44:03.770139 12224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:44:03.770234 12226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:44:03.770275 12218 server_base.cc:1034] running on GCE node
W20260928 04:44:03.770237 12223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:44:03.770624 12218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:44:03.770918 12218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:44:03.772094 12218 hybrid_clock.cc:648] HybridClock initialized: now 1790570643772082 us; error 34 us; skew 500 ppm
Sep 28 04:44:03 dist-test-slave-j5c3 krb5kdc[12111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570643, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:44:03.774863 12218 init.cc:377] Logged in from keytab as kudu/127.1.226.193@KRBTEST.COM (short username kudu)
I20260928 04:44:03.775364 12218 webserver.cc:460] Webserver started at http://127.1.226.193:45727/ using document root <none> and password file <none>
I20260928 04:44:03.775568 12218 fs_manager.cc:359] Metadata directory not provided
I20260928 04:44:03.775610 12218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:44:03.775704 12218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:44:03.776855 12218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/data/instance:
uuid: "102f290edc9d4124b8b4e7eca10da1ad"
format_stamp: "Formatted at 2026-09-28 04:44:03 on dist-test-slave-j5c3"
server_key: "fa6b83b4fe97afeecb9a6c9fce724107"
server_key_iv: "181e9895cb1e6e9e54558eb4da734ae9"
server_key_version: "encryptionkey@0"
I20260928 04:44:03.777171 12218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance:
uuid: "102f290edc9d4124b8b4e7eca10da1ad"
format_stamp: "Formatted at 2026-09-28 04:44:03 on dist-test-slave-j5c3"
server_key: "fa6b83b4fe97afeecb9a6c9fce724107"
server_key_iv: "181e9895cb1e6e9e54558eb4da734ae9"
server_key_version: "encryptionkey@0"
I20260928 04:44:03.778512 12218 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:44:03.779261 12233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:03.779464 12218 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:44:03.779572 12218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "102f290edc9d4124b8b4e7eca10da1ad"
format_stamp: "Formatted at 2026-09-28 04:44:03 on dist-test-slave-j5c3"
server_key: "fa6b83b4fe97afeecb9a6c9fce724107"
server_key_iv: "181e9895cb1e6e9e54558eb4da734ae9"
server_key_version: "encryptionkey@0"
I20260928 04:44:03.779651 12218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:44:03.824110 12218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:44:03.826577 12218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:44:03.826751 12218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:44:03.827000 12218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:44:03.827335 12218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:44:03.827389 12218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:03.827425 12218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:44:03.827452 12218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:03.833316 12218 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:36423
I20260928 04:44:03.833360 12346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:36423 every 8 connection(s)
I20260928 04:44:03.833735 12218 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
I20260928 04:44:03.834784 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 12218
I20260928 04:44:03.834892 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance
I20260928 04:44:03.835090 1931 external_mini_cluster.cc:1428] Setting key d041a99ed4bd85c4e1b046b5e4586b2d
Sep 28 04:44:03 dist-test-slave-j5c3 krb5kdc[12111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570643, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.226.194@KRBTEST.COM; defaulting to no policy
I20260928 04:44:03.844686 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:03.834480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:60949 (local address 127.1.226.254:33561)
0928 04:44:03.834565 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:03.834567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:03.834814 (+ 247us) server_negotiation.cc:407] Connection header received
0928 04:44:03.835561 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:03.835562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:03.835572 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:03.835620 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:44:03.836963 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:03.837465 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:03.839014 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:03.839216 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:03.839275 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:03.841809 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:44:03.841810 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:44:03.841811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:44:03.841818 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:44:03.843227 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:44:03.843660 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:44:03.843661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:44:03.843661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:44:03.843729 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:44:03.843993 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:44:03.843994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:44:03.843994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:44:03.844112 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:44:03.844207 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:03.844499 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 04:44:03.844590 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 04:44:03.844892 12349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:03.834436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:33561 (local address 127.1.226.193:60949)
0928 04:44:03.834635 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 04:44:03.834693 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 04:44:03.835446 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:44:03.835688 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:44:03.835692 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:44:03.836574 (+ 882us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:44:03.836857 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:03.836862 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:03.837565 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:03.837566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:03.838917 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:03.838920 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:03.839306 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:03.839307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:03.839463 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:03.840015 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:44:03.840026 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:44:03.841699 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:44:03.843330 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:44:03.843337 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:44:03.843339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:44:03.843598 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:44:03.843800 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:44:03.843800 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:44:03.843801 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:44:03.843916 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:44:03.844200 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:44:03.844207 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:44:03.844360 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 04:44:03.844476 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 04:44:03.844640 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":99,"threads_started":1}
I20260928 04:44:03.845379 12347 heartbeater.cc:347] Connected to a master server at 127.1.226.254:33561
I20260928 04:44:03.845600 12347 heartbeater.cc:464] Registering TS with master...
I20260928 04:44:03.845948 12347 heartbeater.cc:511] Master 127.1.226.254:33561 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.194@KRBTEST.COM" created.
I20260928 04:44:03.846518 12156 ts_manager.cc:194] Registered new tserver with Master: 102f290edc9d4124b8b4e7eca10da1ad (127.1.226.193:36423)
I20260928 04:44:03.847152 12156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.193@KRBTEST.COM'} at 127.1.226.193:60949
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
I20260928 04:44:03.863960 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:0
--local_ip_for_outbound_sockets=127.1.226.194
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:33561
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:39303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:44:03.948450 12352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:44:03.948679 12352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:44:03.948705 12352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:44:03.949947 12352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:44:03.949993 12352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:44:03.950019 12352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:44:03.951432 12352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:39303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:33561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:44:03.951633 12352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:44:03.951833 12352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:44:03.957032 12360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:44:03.957043 12358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:44:03.957041 12357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:44:03.957392 12352 server_base.cc:1034] running on GCE node
I20260928 04:44:03.957600 12352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:44:03.957863 12352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:44:03.959025 12352 hybrid_clock.cc:648] HybridClock initialized: now 1790570643959005 us; error 39 us; skew 500 ppm
Sep 28 04:44:03 dist-test-slave-j5c3 krb5kdc[12111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570643, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:44:03.961916 12352 init.cc:377] Logged in from keytab as kudu/127.1.226.194@KRBTEST.COM (short username kudu)
I20260928 04:44:03.962401 12352 webserver.cc:460] Webserver started at http://127.1.226.194:39367/ using document root <none> and password file <none>
I20260928 04:44:03.962648 12352 fs_manager.cc:359] Metadata directory not provided
I20260928 04:44:03.962693 12352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:44:03.962867 12352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:44:03.963992 12352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/data/instance:
uuid: "d0ab55388b054a35af42080751a32ab3"
format_stamp: "Formatted at 2026-09-28 04:44:03 on dist-test-slave-j5c3"
server_key: "99b41e23b834012b473cb7c0b1f7c44a"
server_key_iv: "c83b3c4443aa1d9cbbb562f65a5aa662"
server_key_version: "encryptionkey@0"
I20260928 04:44:03.964416 12352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance:
uuid: "d0ab55388b054a35af42080751a32ab3"
format_stamp: "Formatted at 2026-09-28 04:44:03 on dist-test-slave-j5c3"
server_key: "99b41e23b834012b473cb7c0b1f7c44a"
server_key_iv: "c83b3c4443aa1d9cbbb562f65a5aa662"
server_key_version: "encryptionkey@0"
I20260928 04:44:03.965915 12352 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:44:03.966823 12367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:03.967064 12352 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:44:03.967147 12352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "d0ab55388b054a35af42080751a32ab3"
format_stamp: "Formatted at 2026-09-28 04:44:03 on dist-test-slave-j5c3"
server_key: "99b41e23b834012b473cb7c0b1f7c44a"
server_key_iv: "c83b3c4443aa1d9cbbb562f65a5aa662"
server_key_version: "encryptionkey@0"
I20260928 04:44:03.967260 12352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:44:03.986835 12352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:44:03.989326 12352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:44:03.989518 12352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:44:03.989828 12352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:44:03.990168 12352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:44:03.990207 12352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:03.990280 12352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:44:03.990309 12352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:03.996112 12352 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:32903
I20260928 04:44:03.996225 12480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:32903 every 8 connection(s)
I20260928 04:44:03.996563 12352 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
I20260928 04:44:03.999773 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 12352
I20260928 04:44:03.999915 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance
I20260928 04:44:04.000103 1931 external_mini_cluster.cc:1428] Setting key b39e3409921e2b016d169dea9bddee60
Sep 28 04:44:03 dist-test-slave-j5c3 krb5kdc[12111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570643, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:44:04.007645 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:03.997353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:36609 (local address 127.1.226.254:33561)
0928 04:44:03.997442 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:03.997445 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:03.997590 (+ 145us) server_negotiation.cc:407] Connection header received
0928 04:44:03.998130 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:03.998131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:03.998143 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:03.998227 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:44:03.999138 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:03.999752 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.001262 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.001580 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.001658 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:04.004277 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:44:04.004279 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:44:04.004279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:44:04.004288 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:44:04.006087 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:44:04.006572 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:44:04.006575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:44:04.006576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:44:04.006641 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:44:04.006931 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:44:04.006932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:44:04.006933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:44:04.007063 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:44:04.007144 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:04.007449 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 04:44:04.007550 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 04:44:04.007898 12483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:03.997308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:33561 (local address 127.1.226.194:36609)
0928 04:44:03.997461 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 04:44:03.997504 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 04:44:03.998034 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:44:03.998286 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:44:03.998289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:44:03.998857 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:44:03.999047 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:03.999052 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:03.999884 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:03.999885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:04.001152 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:04.001154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.001664 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.001665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:04.001800 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:04.002428 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:44:04.002437 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:44:04.004080 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:44:04.006197 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:44:04.006207 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:44:04.006210 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:44:04.006469 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:44:04.006735 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:44:04.006736 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:44:04.006736 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:44:04.006862 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:44:04.007134 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:44:04.007141 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:44:04.007309 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 04:44:04.007429 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 04:44:04.007601 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260928 04:44:04.008402 12481 heartbeater.cc:347] Connected to a master server at 127.1.226.254:33561
I20260928 04:44:04.008517 12481 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.1.226.195@KRBTEST.COM; defaulting to no policy
I20260928 04:44:04.008745 12481 heartbeater.cc:511] Master 127.1.226.254:33561 requested a full tablet report, sending...
I20260928 04:44:04.009204 12156 ts_manager.cc:194] Registered new tserver with Master: d0ab55388b054a35af42080751a32ab3 (127.1.226.194:32903)
I20260928 04:44:04.009620 12156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.194@KRBTEST.COM'} at 127.1.226.194:36609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
I20260928 04:44:04.027216 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:0
--local_ip_for_outbound_sockets=127.1.226.195
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:33561
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:39303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:44:04.108439 12486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:44:04.108650 12486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:44:04.108673 12486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:44:04.109768 12486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:44:04.109802 12486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:44:04.109827 12486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:44:04.111203 12486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:39303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:33561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:44:04.111394 12486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:44:04.111574 12486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:44:04.116962 12492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:44:04.117019 12494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:44:04.116937 12491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:44:04.117695 12486 server_base.cc:1034] running on GCE node
I20260928 04:44:04.117874 12486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:44:04.118080 12486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:44:04.119262 12486 hybrid_clock.cc:648] HybridClock initialized: now 1790570644119250 us; error 39 us; skew 500 ppm
Sep 28 04:44:04 dist-test-slave-j5c3 krb5kdc[12111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570644, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:44:04.122051 12486 init.cc:377] Logged in from keytab as kudu/127.1.226.195@KRBTEST.COM (short username kudu)
I20260928 04:44:04.122545 12486 webserver.cc:460] Webserver started at http://127.1.226.195:45635/ using document root <none> and password file <none>
I20260928 04:44:04.122795 12486 fs_manager.cc:359] Metadata directory not provided
I20260928 04:44:04.122864 12486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:44:04.123013 12486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:44:04.124084 12486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/data/instance:
uuid: "bfc5f261bf51474c9b0b054619d3c5b0"
format_stamp: "Formatted at 2026-09-28 04:44:04 on dist-test-slave-j5c3"
server_key: "949bb3c75db49417b8d5632f0505c394"
server_key_iv: "7dc0cca64eb09d4d9da3c89c42d99ee1"
server_key_version: "encryptionkey@0"
I20260928 04:44:04.124518 12486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance:
uuid: "bfc5f261bf51474c9b0b054619d3c5b0"
format_stamp: "Formatted at 2026-09-28 04:44:04 on dist-test-slave-j5c3"
server_key: "949bb3c75db49417b8d5632f0505c394"
server_key_iv: "7dc0cca64eb09d4d9da3c89c42d99ee1"
server_key_version: "encryptionkey@0"
I20260928 04:44:04.125900 12486 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:44:04.126673 12501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:04.126888 12486 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:44:04.126988 12486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "bfc5f261bf51474c9b0b054619d3c5b0"
format_stamp: "Formatted at 2026-09-28 04:44:04 on dist-test-slave-j5c3"
server_key: "949bb3c75db49417b8d5632f0505c394"
server_key_iv: "7dc0cca64eb09d4d9da3c89c42d99ee1"
server_key_version: "encryptionkey@0"
I20260928 04:44:04.127070 12486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:44:04.141227 12486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:44:04.143740 12486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:44:04.143908 12486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:44:04.144194 12486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:44:04.144546 12486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:44:04.144599 12486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:04.144642 12486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:44:04.144670 12486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:04.150835 12486 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:42093
I20260928 04:44:04.150873 12614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:42093 every 8 connection(s)
I20260928 04:44:04.151238 12486 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
I20260928 04:44:04.152412 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 12486
I20260928 04:44:04.152525 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance
I20260928 04:44:04.152751 1931 external_mini_cluster.cc:1428] Setting key beb199ed779ebe3d92ff49052f2fe9be
Sep 28 04:44:04 dist-test-slave-j5c3 krb5kdc[12111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570644, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:44:04.161583 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:04.152361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:50427 (local address 127.1.226.254:33561)
0928 04:44:04.152423 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:04.152426 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:04.152697 (+ 271us) server_negotiation.cc:407] Connection header received
0928 04:44:04.153373 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:04.153374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:04.153386 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:04.153425 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:44:04.154463 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.154987 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.156041 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.156323 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.156408 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:04.158738 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:44:04.158739 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:44:04.158739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:44:04.158748 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:44:04.160073 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:44:04.160552 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:44:04.160553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:44:04.160554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:44:04.160619 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:44:04.160900 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:44:04.160901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:44:04.160902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:44:04.161026 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:44:04.161099 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:04.161404 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 04:44:04.161488 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 04:44:04.161809 12617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:04.152352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:33561 (local address 127.1.226.195:50427)
0928 04:44:04.152520 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 04:44:04.152577 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 04:44:04.153279 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:44:04.153509 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:44:04.153518 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:44:04.154152 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:44:04.154364 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:04.154368 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.155088 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.155089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:04.155932 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:04.155935 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.156416 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.156417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:04.156556 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:04.157140 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:44:04.157149 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:44:04.158596 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:44:04.160198 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:44:04.160205 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:44:04.160207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:44:04.160468 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:44:04.160697 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:44:04.160698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:44:04.160698 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:44:04.160823 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:44:04.161101 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:44:04.161108 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:44:04.161261 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 04:44:04.161370 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 04:44:04.161538 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260928 04:44:04.162362 12615 heartbeater.cc:347] Connected to a master server at 127.1.226.254:33561
I20260928 04:44:04.162463 12615 heartbeater.cc:464] Registering TS with master...
I20260928 04:44:04.162612 12615 heartbeater.cc:511] Master 127.1.226.254:33561 requested a full tablet report, sending...
I20260928 04:44:04.163061 12156 ts_manager.cc:194] Registered new tserver with Master: bfc5f261bf51474c9b0b054619d3c5b0 (127.1.226.195:42093)
I20260928 04:44:04.163455 12156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.195@KRBTEST.COM'} at 127.1.226.195:50427
I20260928 04:44:04.164523 1931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:44:04.172252 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:04.165844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36748 (local address 127.1.226.254:33561)
0928 04:44:04.165899 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:04.165901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:04.165959 (+ 58us) server_negotiation.cc:407] Connection header received
0928 04:44:04.166002 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:04.166003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:04.166011 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:04.166064 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:44:04.166574 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.167075 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.167863 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.168043 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.168098 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:04.169385 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:44:04.169386 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:44:04.169386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:44:04.169393 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:44:04.170828 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:44:04.171291 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:44:04.171293 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:44:04.171293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:44:04.171365 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:44:04.171626 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:44:04.171627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:44:04.171628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:44:04.171843 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:44:04.171925 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:04.172053 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 04:44:04.172113 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 04:44:04.173825 12156 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:44:04.175380 12156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:44:04.182168 12631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:04.178098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.1.226.194:32903)
0928 04:44:04.178401 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:04.178404 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:04.178413 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:44:04.178451 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:04.178453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:04.178470 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:04.178736 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:44:04.178882 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.179929 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.181233 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.181732 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.181824 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:04.181886 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:04.182044 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 04:44:04.182085 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":87,"threads_started":1}
I20260928 04:44:04.182154 12628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:04.177825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:32903 (local address 127.0.0.1:47146)
0928 04:44:04.178157 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 04:44:04.178167 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:44:04.178233 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:44:04.178569 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:44:04.178570 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:44:04.178573 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:44:04.178643 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:04.178646 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.180032 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.180033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:04.181138 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:04.181139 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.181824 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.181825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:04.181902 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:04.181916 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 04:44:04.182010 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 04:44:04.182060 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":26,"thread_start_us":57,"threads_started":1}
I20260928 04:44:04.182272 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:04.178093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41420 (local address 127.1.226.193:36423)
0928 04:44:04.178439 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:04.178443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:04.178451 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:44:04.178482 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:04.178483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:04.178495 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:04.178592 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:44:04.178794 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.179568 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.181063 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.181811 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.181873 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:04.181928 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:04.182149 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 04:44:04.182194 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":79,"threads_started":1}
I20260928 04:44:04.182324 12629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:04.177990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34472 (local address 127.1.226.195:42093)
0928 04:44:04.178313 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:04.178318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:04.178377 (+ 59us) server_negotiation.cc:407] Connection header received
0928 04:44:04.178426 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:04.178428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:04.178445 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:04.178538 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:44:04.178751 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.179882 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.181077 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.181784 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.182109 (+ 325us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:04.182170 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:04.182192 (+ 22us) server_negotiation.cc:299] Negotiation successful
0928 04:44:04.182229 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":78,"threads_started":1}
I20260928 04:44:04.182451 12627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:04.177829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:42093 (local address 127.0.0.1:34472)
0928 04:44:04.178312 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0928 04:44:04.178319 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 04:44:04.178355 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:44:04.178565 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:44:04.178568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:44:04.178571 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:44:04.178663 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:04.178665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.180010 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.180011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:04.180979 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:04.180981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.181889 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.181890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:04.181954 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:04.181958 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:44:04.181988 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 04:44:04.182060 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":7,"thread_start_us":95,"threads_started":1}
I20260928 04:44:04.182459 12626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:04.177825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:36423 (local address 127.0.0.1:41420)
0928 04:44:04.177991 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 04:44:04.178057 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 04:44:04.178127 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:44:04.178668 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:44:04.178669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:44:04.178670 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:44:04.178716 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:04.178718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.179671 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.179672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:04.180978 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:04.180981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.181957 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.181957 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:04.182016 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:04.182019 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:44:04.182064 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 04:44:04.182365 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":3,"thread_start_us":90,"threads_started":1}
I20260928 04:44:04.183297 12549 tablet_service.cc:1467] Processing CreateTablet for tablet b4e788fe6f43478eb23b2e4597b74b38 (DEFAULT_TABLE table=test-table [id=97b18930f3564d7493ce971415c8964c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:44:04.183393 12281 tablet_service.cc:1467] Processing CreateTablet for tablet b4e788fe6f43478eb23b2e4597b74b38 (DEFAULT_TABLE table=test-table [id=97b18930f3564d7493ce971415c8964c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:44:04.183507 12415 tablet_service.cc:1467] Processing CreateTablet for tablet b4e788fe6f43478eb23b2e4597b74b38 (DEFAULT_TABLE table=test-table [id=97b18930f3564d7493ce971415c8964c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:44:04.183672 12549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4e788fe6f43478eb23b2e4597b74b38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:44:04.183748 12281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4e788fe6f43478eb23b2e4597b74b38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:44:04.183779 12415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4e788fe6f43478eb23b2e4597b74b38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:44:04.186444 12632 tablet_bootstrap.cc:492] T b4e788fe6f43478eb23b2e4597b74b38 P bfc5f261bf51474c9b0b054619d3c5b0: Bootstrap starting.
I20260928 04:44:04.186967 12633 tablet_bootstrap.cc:492] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3: Bootstrap starting.
I20260928 04:44:04.187178 12634 tablet_bootstrap.cc:492] T b4e788fe6f43478eb23b2e4597b74b38 P 102f290edc9d4124b8b4e7eca10da1ad: Bootstrap starting.
I20260928 04:44:04.187222 12632 tablet_bootstrap.cc:654] T b4e788fe6f43478eb23b2e4597b74b38 P bfc5f261bf51474c9b0b054619d3c5b0: Neither blocks nor log segments found. Creating new log.
I20260928 04:44:04.187621 12633 tablet_bootstrap.cc:654] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3: Neither blocks nor log segments found. Creating new log.
I20260928 04:44:04.187773 12634 tablet_bootstrap.cc:654] T b4e788fe6f43478eb23b2e4597b74b38 P 102f290edc9d4124b8b4e7eca10da1ad: Neither blocks nor log segments found. Creating new log.
I20260928 04:44:04.187860 12632 log.cc:824] T b4e788fe6f43478eb23b2e4597b74b38 P bfc5f261bf51474c9b0b054619d3c5b0: Log is configured to *not* fsync() on all Append() calls
I20260928 04:44:04.188292 12633 log.cc:824] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3: Log is configured to *not* fsync() on all Append() calls
I20260928 04:44:04.188405 12634 log.cc:824] T b4e788fe6f43478eb23b2e4597b74b38 P 102f290edc9d4124b8b4e7eca10da1ad: Log is configured to *not* fsync() on all Append() calls
I20260928 04:44:04.188766 12632 tablet_bootstrap.cc:492] T b4e788fe6f43478eb23b2e4597b74b38 P bfc5f261bf51474c9b0b054619d3c5b0: No bootstrap required, opened a new log
I20260928 04:44:04.188843 12632 ts_tablet_manager.cc:1397] T b4e788fe6f43478eb23b2e4597b74b38 P bfc5f261bf51474c9b0b054619d3c5b0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:44:04.189152 12634 tablet_bootstrap.cc:492] T b4e788fe6f43478eb23b2e4597b74b38 P 102f290edc9d4124b8b4e7eca10da1ad: No bootstrap required, opened a new log
I20260928 04:44:04.189219 12634 ts_tablet_manager.cc:1397] T b4e788fe6f43478eb23b2e4597b74b38 P 102f290edc9d4124b8b4e7eca10da1ad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:44:04.190956 12632 raft_consensus.cc:348] T b4e788fe6f43478eb23b2e4597b74b38 P bfc5f261bf51474c9b0b054619d3c5b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102f290edc9d4124b8b4e7eca10da1ad" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36423 } } peers { permanent_uuid: "d0ab55388b054a35af42080751a32ab3" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32903 } } peers { permanent_uuid: "bfc5f261bf51474c9b0b054619d3c5b0" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 42093 } }
I20260928 04:44:04.191133 12632 raft_consensus.cc:374] T b4e788fe6f43478eb23b2e4597b74b38 P bfc5f261bf51474c9b0b054619d3c5b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:44:04.191171 12632 raft_consensus.cc:729] T b4e788fe6f43478eb23b2e4597b74b38 P bfc5f261bf51474c9b0b054619d3c5b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfc5f261bf51474c9b0b054619d3c5b0, State: Initialized, Role: FOLLOWER
I20260928 04:44:04.191150 12634 raft_consensus.cc:348] T b4e788fe6f43478eb23b2e4597b74b38 P 102f290edc9d4124b8b4e7eca10da1ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102f290edc9d4124b8b4e7eca10da1ad" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36423 } } peers { permanent_uuid: "d0ab55388b054a35af42080751a32ab3" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32903 } } peers { permanent_uuid: "bfc5f261bf51474c9b0b054619d3c5b0" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 42093 } }
I20260928 04:44:04.191285 12634 raft_consensus.cc:374] T b4e788fe6f43478eb23b2e4597b74b38 P 102f290edc9d4124b8b4e7eca10da1ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:44:04.191325 12634 raft_consensus.cc:729] T b4e788fe6f43478eb23b2e4597b74b38 P 102f290edc9d4124b8b4e7eca10da1ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 102f290edc9d4124b8b4e7eca10da1ad, State: Initialized, Role: FOLLOWER
I20260928 04:44:04.191305 12632 consensus_queue.cc:260] T b4e788fe6f43478eb23b2e4597b74b38 P bfc5f261bf51474c9b0b054619d3c5b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102f290edc9d4124b8b4e7eca10da1ad" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36423 } } peers { permanent_uuid: "d0ab55388b054a35af42080751a32ab3" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32903 } } peers { permanent_uuid: "bfc5f261bf51474c9b0b054619d3c5b0" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 42093 } }
I20260928 04:44:04.191452 12634 consensus_queue.cc:260] T b4e788fe6f43478eb23b2e4597b74b38 P 102f290edc9d4124b8b4e7eca10da1ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102f290edc9d4124b8b4e7eca10da1ad" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36423 } } peers { permanent_uuid: "d0ab55388b054a35af42080751a32ab3" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32903 } } peers { permanent_uuid: "bfc5f261bf51474c9b0b054619d3c5b0" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 42093 } }
I20260928 04:44:04.191603 12632 ts_tablet_manager.cc:1428] T b4e788fe6f43478eb23b2e4597b74b38 P bfc5f261bf51474c9b0b054619d3c5b0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:44:04.191637 12615 heartbeater.cc:503] Master 127.1.226.254:33561 was elected leader, sending a full tablet report...
I20260928 04:44:04.191696 12634 ts_tablet_manager.cc:1428] T b4e788fe6f43478eb23b2e4597b74b38 P 102f290edc9d4124b8b4e7eca10da1ad: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 04:44:04.191843 12347 heartbeater.cc:503] Master 127.1.226.254:33561 was elected leader, sending a full tablet report...
I20260928 04:44:04.192279 12633 tablet_bootstrap.cc:492] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3: No bootstrap required, opened a new log
I20260928 04:44:04.192351 12633 ts_tablet_manager.cc:1397] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 04:44:04.193980 12633 raft_consensus.cc:348] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102f290edc9d4124b8b4e7eca10da1ad" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36423 } } peers { permanent_uuid: "d0ab55388b054a35af42080751a32ab3" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32903 } } peers { permanent_uuid: "bfc5f261bf51474c9b0b054619d3c5b0" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 42093 } }
I20260928 04:44:04.194095 12633 raft_consensus.cc:374] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:44:04.194116 12633 raft_consensus.cc:729] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0ab55388b054a35af42080751a32ab3, State: Initialized, Role: FOLLOWER
I20260928 04:44:04.194221 12633 consensus_queue.cc:260] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102f290edc9d4124b8b4e7eca10da1ad" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36423 } } peers { permanent_uuid: "d0ab55388b054a35af42080751a32ab3" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32903 } } peers { permanent_uuid: "bfc5f261bf51474c9b0b054619d3c5b0" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 42093 } }
I20260928 04:44:04.194492 12633 ts_tablet_manager.cc:1428] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 04:44:04.194495 12481 heartbeater.cc:503] Master 127.1.226.254:33561 was elected leader, sending a full tablet report...
W20260928 04:44:04.247228 12482 tablet.cc:2369] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:44:04.334507 12348 tablet.cc:2369] T b4e788fe6f43478eb23b2e4597b74b38 P 102f290edc9d4124b8b4e7eca10da1ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:44:04.385279 12640 raft_consensus.cc:482] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:44:04.385427 12640 raft_consensus.cc:504] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102f290edc9d4124b8b4e7eca10da1ad" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36423 } } peers { permanent_uuid: "d0ab55388b054a35af42080751a32ab3" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32903 } } peers { permanent_uuid: "bfc5f261bf51474c9b0b054619d3c5b0" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 42093 } }
I20260928 04:44:04.385891 12640 leader_election.cc:288] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 102f290edc9d4124b8b4e7eca10da1ad (127.1.226.193:36423), bfc5f261bf51474c9b0b054619d3c5b0 (127.1.226.195:42093)
I20260928 04:44:04.389407 12629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:04.386278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:51895 (local address 127.1.226.195:42093)
0928 04:44:04.386317 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:04.386318 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:04.386432 (+ 114us) server_negotiation.cc:407] Connection header received
0928 04:44:04.386463 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:04.386464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:04.386473 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:04.386541 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:44:04.386703 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.387242 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.388622 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.388995 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.389055 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:04.389062 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:04.389253 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 04:44:04.389300 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:44:04.389448 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:04.386151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:60811 (local address 127.1.226.193:36423)
0928 04:44:04.386198 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:04.386200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:04.386212 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:44:04.386318 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:04.386320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:04.386330 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:04.386394 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:44:04.386611 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.387116 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.388653 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.389015 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.389124 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:04.389132 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:04.389304 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 04:44:04.389368 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 04:44:04.389490 12483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:04.386073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:36423 (local address 127.1.226.194:60811)
0928 04:44:04.386138 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 04:44:04.386151 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:44:04.386227 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:44:04.386454 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:44:04.386456 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:44:04.386457 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:44:04.386534 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:04.386536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.387208 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.387209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:04.388494 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:04.388497 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.389140 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.389141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:04.389206 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:04.389209 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:44:04.389285 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 04:44:04.389355 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":5}
I20260928 04:44:04.389503 12641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:04.386116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:42093 (local address 127.1.226.194:51895)
0928 04:44:04.386359 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 04:44:04.386369 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:44:04.386419 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:44:04.386542 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:44:04.386543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:44:04.386544 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:44:04.386627 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:04.386628 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.387317 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.387318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:04.388494 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:04.388497 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.389092 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.389093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:04.389174 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:04.389184 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 04:44:04.389229 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 04:44:04.389280 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":8,"thread_start_us":146,"threads_started":1}
I20260928 04:44:04.389868 12301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4e788fe6f43478eb23b2e4597b74b38" candidate_uuid: "d0ab55388b054a35af42080751a32ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "102f290edc9d4124b8b4e7eca10da1ad" is_pre_election: true
I20260928 04:44:04.389933 12569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4e788fe6f43478eb23b2e4597b74b38" candidate_uuid: "d0ab55388b054a35af42080751a32ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfc5f261bf51474c9b0b054619d3c5b0" is_pre_election: true
I20260928 04:44:04.390053 12569 raft_consensus.cc:2454] T b4e788fe6f43478eb23b2e4597b74b38 P bfc5f261bf51474c9b0b054619d3c5b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0ab55388b054a35af42080751a32ab3 in term 0.
I20260928 04:44:04.390048 12301 raft_consensus.cc:2454] T b4e788fe6f43478eb23b2e4597b74b38 P 102f290edc9d4124b8b4e7eca10da1ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0ab55388b054a35af42080751a32ab3 in term 0.
I20260928 04:44:04.390316 12368 leader_election.cc:302] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfc5f261bf51474c9b0b054619d3c5b0, d0ab55388b054a35af42080751a32ab3; no voters:
I20260928 04:44:04.390483 12640 raft_consensus.cc:2781] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:44:04.390544 12640 raft_consensus.cc:482] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:44:04.390574 12640 raft_consensus.cc:3037] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:44:04.391273 12640 raft_consensus.cc:504] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102f290edc9d4124b8b4e7eca10da1ad" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36423 } } peers { permanent_uuid: "d0ab55388b054a35af42080751a32ab3" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32903 } } peers { permanent_uuid: "bfc5f261bf51474c9b0b054619d3c5b0" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 42093 } }
I20260928 04:44:04.391433 12640 leader_election.cc:288] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3 [CANDIDATE]: Term 1 election: Requested vote from peers 102f290edc9d4124b8b4e7eca10da1ad (127.1.226.193:36423), bfc5f261bf51474c9b0b054619d3c5b0 (127.1.226.195:42093)
I20260928 04:44:04.391592 12301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4e788fe6f43478eb23b2e4597b74b38" candidate_uuid: "d0ab55388b054a35af42080751a32ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "102f290edc9d4124b8b4e7eca10da1ad"
I20260928 04:44:04.391611 12569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4e788fe6f43478eb23b2e4597b74b38" candidate_uuid: "d0ab55388b054a35af42080751a32ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfc5f261bf51474c9b0b054619d3c5b0"
I20260928 04:44:04.391680 12569 raft_consensus.cc:3037] T b4e788fe6f43478eb23b2e4597b74b38 P bfc5f261bf51474c9b0b054619d3c5b0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:44:04.391737 12301 raft_consensus.cc:3037] T b4e788fe6f43478eb23b2e4597b74b38 P 102f290edc9d4124b8b4e7eca10da1ad [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:44:04.392479 12569 raft_consensus.cc:2454] T b4e788fe6f43478eb23b2e4597b74b38 P bfc5f261bf51474c9b0b054619d3c5b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0ab55388b054a35af42080751a32ab3 in term 1.
I20260928 04:44:04.392570 12301 raft_consensus.cc:2454] T b4e788fe6f43478eb23b2e4597b74b38 P 102f290edc9d4124b8b4e7eca10da1ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0ab55388b054a35af42080751a32ab3 in term 1.
I20260928 04:44:04.392642 12368 leader_election.cc:302] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfc5f261bf51474c9b0b054619d3c5b0, d0ab55388b054a35af42080751a32ab3; no voters:
I20260928 04:44:04.392743 12640 raft_consensus.cc:2781] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:44:04.392918 12640 raft_consensus.cc:686] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3 [term 1 LEADER]: Becoming Leader. State: Replica: d0ab55388b054a35af42080751a32ab3, State: Running, Role: LEADER
I20260928 04:44:04.393030 12640 consensus_queue.cc:237] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102f290edc9d4124b8b4e7eca10da1ad" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36423 } } peers { permanent_uuid: "d0ab55388b054a35af42080751a32ab3" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32903 } } peers { permanent_uuid: "bfc5f261bf51474c9b0b054619d3c5b0" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 42093 } }
I20260928 04:44:04.393898 12156 catalog_manager.cc:5495] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0ab55388b054a35af42080751a32ab3 (127.1.226.194). New cstate: current_term: 1 leader_uuid: "d0ab55388b054a35af42080751a32ab3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102f290edc9d4124b8b4e7eca10da1ad" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0ab55388b054a35af42080751a32ab3" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bfc5f261bf51474c9b0b054619d3c5b0" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 42093 } health_report { overall_health: UNKNOWN } } }
W20260928 04:44:04.402055 12616 tablet.cc:2369] T b4e788fe6f43478eb23b2e4597b74b38 P bfc5f261bf51474c9b0b054619d3c5b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:44:04.408511 12631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:04.405997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47154 (local address 127.1.226.194:32903)
0928 04:44:04.406069 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:04.406070 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:04.406094 (+ 24us) server_negotiation.cc:407] Connection header received
0928 04:44:04.406125 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:04.406126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:04.406142 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:04.406209 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:44:04.406439 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.406950 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.407729 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:04.407917 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:04.407971 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:04.408109 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:44:04.408282 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:44:04.408351 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:04.408412 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 04:44:04.408431 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 04:44:04.410418 12569 raft_consensus.cc:1261] T b4e788fe6f43478eb23b2e4597b74b38 P bfc5f261bf51474c9b0b054619d3c5b0 [term 1 FOLLOWER]: Refusing update from remote peer d0ab55388b054a35af42080751a32ab3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:44:04.410439 12301 raft_consensus.cc:1261] T b4e788fe6f43478eb23b2e4597b74b38 P 102f290edc9d4124b8b4e7eca10da1ad [term 1 FOLLOWER]: Refusing update from remote peer d0ab55388b054a35af42080751a32ab3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:44:04.410840 12642 consensus_queue.cc:1035] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "102f290edc9d4124b8b4e7eca10da1ad" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:44:04.410970 12640 consensus_queue.cc:1035] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfc5f261bf51474c9b0b054619d3c5b0" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 42093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:44:04.413509 12645 mvcc.cc:205] Tried to move back new op lower bound from 7334177359501959168 to 7334177359434432512. Current Snapshot: MvccSnapshot[applied={T|T < 7334177359501959168}]
I20260928 04:44:04.413707 12646 mvcc.cc:205] Tried to move back new op lower bound from 7334177359501959168 to 7334177359434432512. Current Snapshot: MvccSnapshot[applied={T|T < 7334177359501959168}]
I20260928 04:44:04.415702 12647 mvcc.cc:205] Tried to move back new op lower bound from 7334177359501959168 to 7334177359434432512. Current Snapshot: MvccSnapshot[applied={T|T < 7334177359501959168 or (T in {7334177359501959168})}]
I20260928 04:44:04.415930 12156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:44:04.416018 12156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:44:04.416831 12156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3482117f03e4bb49a45c0e5a97928ee: Sending DeleteTablet for 3 replicas of tablet b4e788fe6f43478eb23b2e4597b74b38
I20260928 04:44:04.417150 12415 tablet_service.cc:1514] Processing DeleteTablet for tablet b4e788fe6f43478eb23b2e4597b74b38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:44:04 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:47146
I20260928 04:44:04.417160 12281 tablet_service.cc:1514] Processing DeleteTablet for tablet b4e788fe6f43478eb23b2e4597b74b38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:44:04 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:41420
I20260928 04:44:04.417403 12656 tablet_replica.cc:330] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3: stopping tablet replica
I20260928 04:44:04.417389 12549 tablet_service.cc:1514] Processing DeleteTablet for tablet b4e788fe6f43478eb23b2e4597b74b38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:44:04 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:34472
I20260928 04:44:04.417500 12656 raft_consensus.cc:2229] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:44:04.417587 12656 raft_consensus.cc:2258] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:44:04.417604 12657 tablet_replica.cc:330] T b4e788fe6f43478eb23b2e4597b74b38 P bfc5f261bf51474c9b0b054619d3c5b0: stopping tablet replica
I20260928 04:44:04.417747 12655 tablet_replica.cc:330] T b4e788fe6f43478eb23b2e4597b74b38 P 102f290edc9d4124b8b4e7eca10da1ad: stopping tablet replica
I20260928 04:44:04.417867 12655 raft_consensus.cc:2229] T b4e788fe6f43478eb23b2e4597b74b38 P 102f290edc9d4124b8b4e7eca10da1ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:44:04.418004 12655 raft_consensus.cc:2258] T b4e788fe6f43478eb23b2e4597b74b38 P 102f290edc9d4124b8b4e7eca10da1ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:44:04.418051 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 12218
I20260928 04:44:04.418100 12656 ts_tablet_manager.cc:1905] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:44:04.420076 12656 ts_tablet_manager.cc:1918] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:44:04.420141 12656 log.cc:1196] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-1/wal/wals/b4e788fe6f43478eb23b2e4597b74b38
I20260928 04:44:04.420403 12657 raft_consensus.cc:2229] T b4e788fe6f43478eb23b2e4597b74b38 P bfc5f261bf51474c9b0b054619d3c5b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:44:04.420521 12657 raft_consensus.cc:2258] T b4e788fe6f43478eb23b2e4597b74b38 P bfc5f261bf51474c9b0b054619d3c5b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:44:04.420835 12656 ts_tablet_manager.cc:1939] T b4e788fe6f43478eb23b2e4597b74b38 P d0ab55388b054a35af42080751a32ab3: Deleting consensus metadata
I20260928 04:44:04.421648 12143 catalog_manager.cc:4841] TS d0ab55388b054a35af42080751a32ab3 (127.1.226.194:32903): tablet b4e788fe6f43478eb23b2e4597b74b38 (table test-table [id=97b18930f3564d7493ce971415c8964c]) successfully deleted
I20260928 04:44:04.421888 12657 ts_tablet_manager.cc:1905] T b4e788fe6f43478eb23b2e4597b74b38 P bfc5f261bf51474c9b0b054619d3c5b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:44:04.423280 12657 ts_tablet_manager.cc:1918] T b4e788fe6f43478eb23b2e4597b74b38 P bfc5f261bf51474c9b0b054619d3c5b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:44:04.423388 12657 log.cc:1196] T b4e788fe6f43478eb23b2e4597b74b38 P bfc5f261bf51474c9b0b054619d3c5b0: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570585693787-1931-0/minicluster-data/ts-2/wal/wals/b4e788fe6f43478eb23b2e4597b74b38
I20260928 04:44:04.423712 12657 ts_tablet_manager.cc:1939] T b4e788fe6f43478eb23b2e4597b74b38 P bfc5f261bf51474c9b0b054619d3c5b0: Deleting consensus metadata
I20260928 04:44:04.424131 12140 catalog_manager.cc:4841] TS bfc5f261bf51474c9b0b054619d3c5b0 (127.1.226.195:42093): tablet b4e788fe6f43478eb23b2e4597b74b38 (table test-table [id=97b18930f3564d7493ce971415c8964c]) successfully deleted
W20260928 04:44:04.425143 12141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.226.193:36423 (error 108)
I20260928 04:44:04.425371 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 12352
I20260928 04:44:04.426690 12626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:04.426519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:36423 (local address 127.0.0.1:41432)
0928 04:44:04.426601 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 04:44:04.426651 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.193:36423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 04:44:04.426846 12141 catalog_manager.cc:4584] TS 102f290edc9d4124b8b4e7eca10da1ad (127.1.226.193:36423): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b4e788fe6f43478eb23b2e4597b74b38: Network error: Client connection negotiation failed: client connection to 127.1.226.193:36423: connect: Connection refused (error 111)
I20260928 04:44:04.430995 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 12486
I20260928 04:44:04.435376 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 12124
2026-09-28T04:44:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2170 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:44:04 dist-test-slave-j5c3 krb5kdc[12661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:44:04 dist-test-slave-j5c3 krb5kdc[12661](info): set up 2 sockets
Sep 28 04:44:04 dist-test-slave-j5c3 krb5kdc[12661](info): commencing operation
krb5kdc: starting...
W20260928 04:44:05.461202 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:44:05 dist-test-slave-j5c3 krb5kdc[12661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:44:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:44:05Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.226.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.226.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.226.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/rangeradmin_127.1.226.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.226.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/rangeradmin_127.1.226.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.226.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.226.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.226.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/rangerlookup_127.1.226.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.226.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/rangerlookup_127.1.226.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.226.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.226.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.226.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/HTTP_127.1.226.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.226.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/HTTP_127.1.226.212@KRBTEST.COM.keytab.
I20260928 04:44:05.634694 1931 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasky6vTKM/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/postgres -l logfile start
2026-09-28 04:44:08.183 UTC [12691] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 04:44:08.183 UTC [12691] LOG: listening on IPv4 address "127.1.226.212", port 55837
2026-09-28 04:44:08.187 UTC [12691] LOG: listening on Unix socket "/tmp/.s.PGSQL.55837"
2026-09-28 04:44:08.209 UTC [12694] LOG: database system was shut down at 2026-09-28 04:44:06 UTC
2026-09-28 04:44:08.212 UTC [12691] LOG: database system is ready to accept connections
I20260928 04:44:09.185482 1931 mini_postgres.cc:96] Postgres bound to 55837
2026-09-28 04:44:09.191 UTC [12702] FATAL: database "slave" does not exist
127.1.226.212:55837 - accepting connections
I20260928 04:44:09.192276 1931 mini_ranger.cc:161] Starting Ranger...
I20260928 04:44:09.207764 1931 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 04:44:09.688854 1931 mini_ranger.cc:87] Created ranger Postgres database
I20260928 04:44:09.688943 1931 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ranger-admin
2026-09-28 04:44:10,010 [I] DB FLAVOR :POSTGRES
2026-09-28 04:44:10,011 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 04:44:10,011 [I] Checking connection..
2026-09-28 04:44:10,011 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:55837/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 04:44:10,262 [I] Checking connection passed.
2026-09-28 04:44:10,262 [I] --------- Verifying version history table ---------
2026-09-28 04:44:10,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:55837/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 04:44:10,539 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 04:44:10,539 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:55837/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 04:44:10,807 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 04:44:10,807 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 04:44:10,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:55837/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 04:44:11.071 UTC [12766] WARNING: there is no transaction in progress
2026-09-28 04:44:11,078 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 04:44:11,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:55837/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 04:44:11,333 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 04:44:11,333 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 04:44:11,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:55837/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 04:44:11,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:55837/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 04:44:11,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:55837/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j5c3.c.gcp-upstream.internal','N') ;"
2026-09-28 04:44:12,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:55837/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 04:44:12,312 [I] Table x_portal_user does not exist in database ranger
2026-09-28 04:44:12,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:55837/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 04:44:12,575 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 04:44:12,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:55837/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 04:44:12,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:55837/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 04:44:13,074 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 04:44:13,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:55837/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 04:44:14.067 UTC [12874] WARNING: there is no transaction in progress
2026-09-28 04:44:14.082 UTC [12874] WARNING: there is no transaction in progress
2026-09-28 04:44:14.099 UTC [12874] WARNING: there is no transaction in progress
2026-09-28 04:44:14.116 UTC [12874] WARNING: there is no transaction in progress
2026-09-28 04:44:14.132 UTC [12874] WARNING: there is no transaction in progress
2026-09-28 04:44:14.153 UTC [12874] WARNING: there is no transaction in progress
2026-09-28 04:44:14.212 UTC [12874] WARNING: there is no transaction in progress
2026-09-28 04:44:14.220 UTC [12874] WARNING: there is no transaction in progress
2026-09-28 04:44:14.234 UTC [12874] WARNING: there is no transaction in progress
2026-09-28 04:44:14.245 UTC [12874] WARNING: there is no transaction in progress
2026-09-28 04:44:14.256 UTC [12874] WARNING: there is no transaction in progress
2026-09-28 04:44:14.272 UTC [12874] WARNING: there is no transaction in progress
2026-09-28 04:44:14.282 UTC [12874] WARNING: there is no transaction in progress
2026-09-28 04:44:14.289 UTC [12874] WARNING: there is no transaction in progress
2026-09-28 04:44:14.297 UTC [12874] WARNING: there is no transaction in progress
2026-09-28 04:44:14.304 UTC [12874] WARNING: there is no transaction in progress
2026-09-28 04:44:15,029 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 04:44:15,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:55837/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 04:44:15,280 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 04:44:15,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:55837/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 04:44:15,544 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 04:44:15,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:55837/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 04:44:15,785 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 04:44:15,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:55837/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 04:44:16,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:55837/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 04:44:16,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:55837/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j5c3.c.gcp-upstream.internal';"
2026-09-28 04:44:16,513 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 04:44:16,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:55837/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 04:44:16,761 [I] DB_PATCHES have already been applied
I20260928 04:44:16.768007 1931 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ranger-admin:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasky6vTKM/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 04:44:16.768078 1931 mini_ranger.cc:193] Using host: 127.1.226.212
I20260928 04:44:16.771267 1931 mini_ranger.cc:227] Ranger admin URL: http://127.1.226.212:55919
Sep 28, 2026 4:44:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:44:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:44:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 4:44:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 4:44:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 4:44:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 4:44:17 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 04:44:17.947764 1931 mini_ranger.cc:160] Time spent starting Ranger: real 8.755s user 0.001s sys 0.008s
Sep 28, 2026 4:44:38 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 4:44:38 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 4:44:38 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 4:44:38 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 4:44:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 4:44:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 4:44:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 4:44:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 4:44:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 4:44:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 4:44:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 4:44:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 4:44:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 4:44:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 4:44:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 4:44:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 4:44:38 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 4:44:39 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 04:44:40.499994 1931 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.226.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.226.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.226.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/rangerkms_127.1.226.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.226.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/rangerkms_127.1.226.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.226.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.226.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.226.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/HTTP_127.1.226.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.226.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/HTTP_127.1.226.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 04:44:40 dist-test-slave-j5c3 krb5kdc[12661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570680, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 04:44:40.586475 1931 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 04:44:40.596863 1931 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 04:44:41.644258 1931 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 04:44:41.644353 1931 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ranger-kms
I20260928 04:44:41.644367 1931 mini_ranger_kms.cc:227] Using postgres at 127.1.226.212:55837
2026-09-28 04:44:41,828 [I] DB FLAVOR :POSTGRES
2026-09-28 04:44:41,828 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 04:44:41,829 [I] Checking connection
2026-09-28 04:44:41,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:55837/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 04:44:42,090 [I] connection success
2026-09-28 04:44:42,091 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 04:44:42,091 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 04:44:42,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:55837/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 04:44:42,331 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 04:44:42,332 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 04:44:42,332 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 04:44:42,332 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:55837/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 04:44:42,593 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 04:44:43.103659 1931 mini_ranger_kms.cc:310] Created kms service
I20260928 04:44:43.324132 1931 mini_ranger_kms.cc:326] Created kudu user
I20260928 04:44:43.370582 1931 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 04:44:43 dist-test-slave-j5c3 krb5kdc[12661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570680, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.226.212@KRBTEST.COM
I20260928 04:44:43.814713 1931 mini_ranger_kms.cc:384] Added ranger policy
I20260928 04:44:43.814828 1931 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ranger-kms:/tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 04:44:43.814851 1931 mini_ranger_kms.cc:240] Using host: 127.1.226.212
I20260928 04:44:43.818077 1931 mini_ranger_kms.cc:276] Ranger KMS PID: 13055
I20260928 04:44:43.818205 1931 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.226.212:47077
Sep 28, 2026 4:44:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:44:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:44:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 4:44:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 4:44:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 4:44:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 04:44:46.114481 1931 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.528s user 0.016s sys 0.151s
I20260928 04:44:46.114593 1931 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 04:44:46.114692 1931 mini_ranger_kms.cc:401] 127.1.226.212:47077/kms/v1/keys
Sep 28 04:44:46 dist-test-slave-j5c3 krb5kdc[12661](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790570686, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.226.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 04:44:46 dist-test-slave-j5c3 krb5kdc[12661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570686, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.226.212@KRBTEST.COM for HTTP/127.1.226.212@KRBTEST.COM
Sep 28 04:44:47 dist-test-slave-j5c3 krb5kdc[12661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570686, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.226.212@KRBTEST.COM for HTTP/127.1.226.212@KRBTEST.COM
Sep 28 04:44:49 dist-test-slave-j5c3 krb5kdc[12661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:44:49.316396 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:38995
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:32781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:38995
--ranger_config_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.226.212:47077/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:44:49.400543 13105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:44:49.400837 13105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:44:49.400866 13105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:44:49.402078 13105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:44:49.402120 13105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:44:49.402134 13105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:44:49.402148 13105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:44:49.402197 13105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:44:49.403723 13105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:32781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.226.212:47077/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:38995
--ranger_config_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:38995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:44:49.403968 13105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:44:49.404228 13105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:44:49.409397 13110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:44:49.409842 13113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:44:49.409438 13111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:44:49.409564 13105 server_base.cc:1034] running on GCE node
I20260928 04:44:49.410305 13105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:44:49.410687 13105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:44:49.411880 13105 hybrid_clock.cc:648] HybridClock initialized: now 1790570689411878 us; error 48 us; skew 500 ppm
Sep 28 04:44:49 dist-test-slave-j5c3 krb5kdc[12661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570689, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:44:49.418195 13105 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:44:49.418747 13105 webserver.cc:460] Webserver started at http://127.1.226.254:44383/ using document root <none> and password file <none>
I20260928 04:44:49.419003 13105 fs_manager.cc:359] Metadata directory not provided
I20260928 04:44:49.419070 13105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:44:49.419257 13105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 04:44:49 dist-test-slave-j5c3 krb5kdc[12661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570689, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for HTTP/127.1.226.212@KRBTEST.COM
I20260928 04:44:49.474720 13105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "f0d7d50620384a0b8c267c226c7f4e17"
format_stamp: "Formatted at 2026-09-28 04:44:49 on dist-test-slave-j5c3"
server_key: "d8e44ec0f0b7a72554eb2933acdd2007"
server_key_iv: "75ee6f7ada3eefa10548f82f0819313d"
server_key_version: "kuduclusterkey@0"
I20260928 04:44:49.475257 13105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "f0d7d50620384a0b8c267c226c7f4e17"
format_stamp: "Formatted at 2026-09-28 04:44:49 on dist-test-slave-j5c3"
server_key: "d8e44ec0f0b7a72554eb2933acdd2007"
server_key_iv: "75ee6f7ada3eefa10548f82f0819313d"
server_key_version: "kuduclusterkey@0"
I20260928 04:44:49.477294 13105 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 04:44:49.489176 13120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:49.489447 13105 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:44:49.489548 13105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "f0d7d50620384a0b8c267c226c7f4e17"
format_stamp: "Formatted at 2026-09-28 04:44:49 on dist-test-slave-j5c3"
server_key: "d8e44ec0f0b7a72554eb2933acdd2007"
server_key_iv: "75ee6f7ada3eefa10548f82f0819313d"
server_key_version: "kuduclusterkey@0"
I20260928 04:44:49.489670 13105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:44:49.503225 13105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:44:49.505254 13105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:44:49.505427 13105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:44:49.510644 13105 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:38995
I20260928 04:44:49.510664 13172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:38995 every 8 connection(s)
I20260928 04:44:49.511077 13105 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:44:49.511893 13173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:44:49.512557 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 13105
I20260928 04:44:49.512665 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:44:49.512856 1931 external_mini_cluster.cc:1428] Setting key f2ce64eada9d8d0f7ec1031986f70a2d
I20260928 04:44:49.515148 13173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0d7d50620384a0b8c267c226c7f4e17: Bootstrap starting.
I20260928 04:44:49.515872 13173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0d7d50620384a0b8c267c226c7f4e17: Neither blocks nor log segments found. Creating new log.
I20260928 04:44:49.516306 13173 log.cc:824] T 00000000000000000000000000000000 P f0d7d50620384a0b8c267c226c7f4e17: Log is configured to *not* fsync() on all Append() calls
I20260928 04:44:49.517105 13173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0d7d50620384a0b8c267c226c7f4e17: No bootstrap required, opened a new log
Sep 28 04:44:49 dist-test-slave-j5c3 krb5kdc[12661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:44:49.519603 13173 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0d7d50620384a0b8c267c226c7f4e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d7d50620384a0b8c267c226c7f4e17" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38995 } }
I20260928 04:44:49.519802 13173 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0d7d50620384a0b8c267c226c7f4e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:44:49.519871 13173 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0d7d50620384a0b8c267c226c7f4e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0d7d50620384a0b8c267c226c7f4e17, State: Initialized, Role: FOLLOWER
I20260928 04:44:49.520022 13173 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0d7d50620384a0b8c267c226c7f4e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d7d50620384a0b8c267c226c7f4e17" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38995 } }
I20260928 04:44:49.520120 13173 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0d7d50620384a0b8c267c226c7f4e17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:44:49.520215 13173 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0d7d50620384a0b8c267c226c7f4e17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:44:49.520270 13173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0d7d50620384a0b8c267c226c7f4e17 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:44:49.521066 13173 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0d7d50620384a0b8c267c226c7f4e17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d7d50620384a0b8c267c226c7f4e17" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38995 } }
I20260928 04:44:49.521183 13173 leader_election.cc:302] T 00000000000000000000000000000000 P f0d7d50620384a0b8c267c226c7f4e17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0d7d50620384a0b8c267c226c7f4e17; no voters:
I20260928 04:44:49.521399 13173 leader_election.cc:288] T 00000000000000000000000000000000 P f0d7d50620384a0b8c267c226c7f4e17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:44:49.521461 13178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0d7d50620384a0b8c267c226c7f4e17 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:44:49.521664 13178 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0d7d50620384a0b8c267c226c7f4e17 [term 1 LEADER]: Becoming Leader. State: Replica: f0d7d50620384a0b8c267c226c7f4e17, State: Running, Role: LEADER
I20260928 04:44:49.521757 13173 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0d7d50620384a0b8c267c226c7f4e17 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:44:49.522140 13178 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0d7d50620384a0b8c267c226c7f4e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d7d50620384a0b8c267c226c7f4e17" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38995 } }
I20260928 04:44:49.522745 13178 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0d7d50620384a0b8c267c226c7f4e17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0d7d50620384a0b8c267c226c7f4e17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d7d50620384a0b8c267c226c7f4e17" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38995 } } }
I20260928 04:44:49.523391 13178 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0d7d50620384a0b8c267c226c7f4e17 [sys.catalog]: This master's current role is: LEADER
I20260928 04:44:49.523447 13176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:49.513391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42132 (local address 127.1.226.254:38995)
0928 04:44:49.513575 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:49.513586 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:49.513617 (+ 31us) server_negotiation.cc:407] Connection header received
0928 04:44:49.514035 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:49.514042 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:49.514090 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:49.514243 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:44:49.514874 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:49.515840 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:49.516848 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:49.517327 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:49.517443 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:49.519799 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:44:49.519804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:44:49.519804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:44:49.519819 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:44:49.521758 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:44:49.522143 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:44:49.522147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:44:49.522147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:44:49.522255 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:44:49.522461 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:44:49.522462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:44:49.522462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:44:49.522781 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:44:49.522929 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:49.523060 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 04:44:49.523176 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":67,"threads_started":1}
I20260928 04:44:49.523262 13173 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 04:44:49.524175 13180 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0d7d50620384a0b8c267c226c7f4e17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0d7d50620384a0b8c267c226c7f4e17. Latest consensus state: current_term: 1 leader_uuid: "f0d7d50620384a0b8c267c226c7f4e17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d7d50620384a0b8c267c226c7f4e17" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38995 } } }
I20260928 04:44:49.524462 13180 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0d7d50620384a0b8c267c226c7f4e17 [sys.catalog]: This master's current role is: LEADER
I20260928 04:44:49.524333 13193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:44:49.525082 13193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:44:49.533071 13193 catalog_manager.cc:1347] Generated new cluster ID: 996b850e386b4139b4d4761761209935
I20260928 04:44:49.533159 13193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:44:49.552078 13193 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:44:49.552964 13193 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:44:49.559243 13193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0d7d50620384a0b8c267c226c7f4e17: Generated new TSK 0
I20260928 04:44:49.559561 13193 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.226.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
I20260928 04:44:50.300132 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.226.212:47077/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:0
--local_ip_for_outbound_sockets=127.1.226.193
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:38995
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:32781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:44:50.432658 13218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:44:50.432960 13218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:44:50.432998 13218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:44:50.434717 13218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:44:50.434793 13218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:44:50.434844 13218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:44:50.437531 13218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:32781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.226.212:47077/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:38995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:44:50.437815 13218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:44:50.438127 13218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:44:50.444941 13223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:44:50.445142 13224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:44:50.445379 13226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:44:50.445485 13218 server_base.cc:1034] running on GCE node
I20260928 04:44:50.445704 13218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:44:50.446066 13218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:44:50.447259 13218 hybrid_clock.cc:648] HybridClock initialized: now 1790570690447236 us; error 50 us; skew 500 ppm
Sep 28 04:44:50 dist-test-slave-j5c3 krb5kdc[12661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570690, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:44:50.450376 13218 init.cc:377] Logged in from keytab as kudu/127.1.226.193@KRBTEST.COM (short username kudu)
I20260928 04:44:50.450985 13218 webserver.cc:460] Webserver started at http://127.1.226.193:34427/ using document root <none> and password file <none>
I20260928 04:44:50.451292 13218 fs_manager.cc:359] Metadata directory not provided
I20260928 04:44:50.451373 13218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:44:50.451560 13218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 04:44:50 dist-test-slave-j5c3 krb5kdc[12661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570690, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for HTTP/127.1.226.212@KRBTEST.COM
I20260928 04:44:50.465926 13218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-0/data/instance:
uuid: "157eafdd618e42ee9886f73244d9d0f4"
format_stamp: "Formatted at 2026-09-28 04:44:50 on dist-test-slave-j5c3"
server_key: "af815dd728014376245f266000a9570e"
server_key_iv: "fa14257bd6fdee5b04ac594d9454e1b9"
server_key_version: "kuduclusterkey@0"
I20260928 04:44:50.466598 13218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance:
uuid: "157eafdd618e42ee9886f73244d9d0f4"
format_stamp: "Formatted at 2026-09-28 04:44:50 on dist-test-slave-j5c3"
server_key: "af815dd728014376245f266000a9570e"
server_key_iv: "fa14257bd6fdee5b04ac594d9454e1b9"
server_key_version: "kuduclusterkey@0"
I20260928 04:44:50.470013 13218 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 04:44:50.488389 13233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:50.488703 13218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 04:44:50.488814 13218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "157eafdd618e42ee9886f73244d9d0f4"
format_stamp: "Formatted at 2026-09-28 04:44:50 on dist-test-slave-j5c3"
server_key: "af815dd728014376245f266000a9570e"
server_key_iv: "fa14257bd6fdee5b04ac594d9454e1b9"
server_key_version: "kuduclusterkey@0"
I20260928 04:44:50.488951 13218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:44:50.515089 13218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:44:50.517860 13218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:44:50.518033 13218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:44:50.518325 13218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:44:50.518807 13218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:44:50.518850 13218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:50.518874 13218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:44:50.518887 13218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:50.526930 13218 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:45803
I20260928 04:44:50.527523 13218 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
I20260928 04:44:50.528184 13348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:45803 every 8 connection(s)
I20260928 04:44:50.529031 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 13218
I20260928 04:44:50.529167 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance
I20260928 04:44:50.529441 1931 external_mini_cluster.cc:1428] Setting key 85ab77fd022b695c0e750c4a2a837d24
Sep 28 04:44:50 dist-test-slave-j5c3 krb5kdc[12661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570690, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.226.194@KRBTEST.COM; defaulting to no policy
I20260928 04:44:50.543483 13352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:50.529384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:33771 (local address 127.1.226.254:38995)
0928 04:44:50.529528 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:50.529531 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:50.530863 (+ 1332us) server_negotiation.cc:407] Connection header received
0928 04:44:50.531485 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:50.531486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:50.531501 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:50.531581 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:44:50.532501 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:50.533416 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:50.535393 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:50.535874 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:50.535940 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:50.539549 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:44:50.539552 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:44:50.539553 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:44:50.539565 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:44:50.541661 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:44:50.542250 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:44:50.542251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:44:50.542252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:44:50.542328 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:44:50.542614 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:44:50.542615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:44:50.542616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:44:50.542759 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:44:50.542849 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:50.543254 (+ 405us) server_negotiation.cc:299] Negotiation successful
0928 04:44:50.543365 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
I20260928 04:44:50.543725 13351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:50.529006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38995 (local address 127.1.226.193:33771)
0928 04:44:50.530715 (+ 1709us) negotiation.cc:107] Waiting for socket to connect
0928 04:44:50.530760 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 04:44:50.531340 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:44:50.531669 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:44:50.531676 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:44:50.531993 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:44:50.532366 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:50.532373 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:50.533555 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:50.533556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:50.535244 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:50.535248 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:50.536115 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:50.536117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:50.536374 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:50.537238 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:44:50.537252 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:44:50.539364 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:44:50.541782 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:44:50.541794 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:44:50.541797 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:44:50.542142 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:44:50.542395 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:44:50.542396 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:44:50.542397 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:44:50.542523 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:44:50.542839 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:44:50.542848 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:44:50.543056 (+ 208us) client_negotiation.cc:769] Sending connection context
0928 04:44:50.543227 (+ 171us) client_negotiation.cc:240] Negotiation successful
0928 04:44:50.543422 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1660,"thread_start_us":93,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.194@KRBTEST.COM" created.
I20260928 04:44:50.544389 13349 heartbeater.cc:347] Connected to a master server at 127.1.226.254:38995
I20260928 04:44:50.544505 13349 heartbeater.cc:464] Registering TS with master...
I20260928 04:44:50.544780 13349 heartbeater.cc:511] Master 127.1.226.254:38995 requested a full tablet report, sending...
I20260928 04:44:50.545483 13137 ts_manager.cc:194] Registered new tserver with Master: 157eafdd618e42ee9886f73244d9d0f4 (127.1.226.193:45803)
I20260928 04:44:50.546523 13137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.193@KRBTEST.COM'} at 127.1.226.193:33771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
I20260928 04:44:50.563395 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.226.212:47077/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:0
--local_ip_for_outbound_sockets=127.1.226.194
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:38995
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:32781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:44:50.680770 13355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:44:50.681080 13355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:44:50.681120 13355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:44:50.682922 13355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:44:50.682987 13355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:44:50.683032 13355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:44:50.685333 13355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:32781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.226.212:47077/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:38995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:44:50.685600 13355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:44:50.685993 13355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:44:50.692445 13365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:44:50.692606 13363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:44:50.692749 13362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:44:50.693189 13355 server_base.cc:1034] running on GCE node
I20260928 04:44:50.693397 13355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:44:50.693684 13355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:44:50.694876 13355 hybrid_clock.cc:648] HybridClock initialized: now 1790570690694853 us; error 39 us; skew 500 ppm
Sep 28 04:44:50 dist-test-slave-j5c3 krb5kdc[12661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570690, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:44:50.698273 13355 init.cc:377] Logged in from keytab as kudu/127.1.226.194@KRBTEST.COM (short username kudu)
I20260928 04:44:50.698839 13355 webserver.cc:460] Webserver started at http://127.1.226.194:34587/ using document root <none> and password file <none>
I20260928 04:44:50.699105 13355 fs_manager.cc:359] Metadata directory not provided
I20260928 04:44:50.699167 13355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:44:50.699352 13355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 04:44:50 dist-test-slave-j5c3 krb5kdc[12661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570690, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for HTTP/127.1.226.212@KRBTEST.COM
Sep 28 04:44:50 dist-test-slave-j5c3 krb5kdc[12661](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790570690, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:44:50.720245 13355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-1/data/instance:
uuid: "16e324bdac7e4c148f496ba72e71480a"
format_stamp: "Formatted at 2026-09-28 04:44:50 on dist-test-slave-j5c3"
server_key: "9a2de3d51d08cbf7b8b51bde1b414eff"
server_key_iv: "f2052cfa2b49387744f24aa3de97bc46"
server_key_version: "kuduclusterkey@0"
I20260928 04:44:50.720717 13355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance:
uuid: "16e324bdac7e4c148f496ba72e71480a"
format_stamp: "Formatted at 2026-09-28 04:44:50 on dist-test-slave-j5c3"
server_key: "9a2de3d51d08cbf7b8b51bde1b414eff"
server_key_iv: "f2052cfa2b49387744f24aa3de97bc46"
server_key_version: "kuduclusterkey@0"
I20260928 04:44:50.722718 13355 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 04:44:50.739468 13372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:50.739714 13355 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 04:44:50.739823 13355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "16e324bdac7e4c148f496ba72e71480a"
format_stamp: "Formatted at 2026-09-28 04:44:50 on dist-test-slave-j5c3"
server_key: "9a2de3d51d08cbf7b8b51bde1b414eff"
server_key_iv: "f2052cfa2b49387744f24aa3de97bc46"
server_key_version: "kuduclusterkey@0"
I20260928 04:44:50.739929 13355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:44:50.777473 13355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:44:50.780284 13355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:44:50.780522 13355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:44:50.780830 13355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:44:50.781297 13355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:44:50.781389 13355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:50.781504 13355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:44:50.781577 13355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:50.788610 13355 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:40043
I20260928 04:44:50.788959 13486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:40043 every 8 connection(s)
I20260928 04:44:50.789180 13355 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
I20260928 04:44:50.792407 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 13355
I20260928 04:44:50.792518 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance
I20260928 04:44:50.792714 1931 external_mini_cluster.cc:1428] Setting key b007c9ff3722e1dd929f31f4316b64d5
Sep 28 04:44:50 dist-test-slave-j5c3 krb5kdc[12661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570690, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:44:50.802341 13352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:50.790378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:44291 (local address 127.1.226.254:38995)
0928 04:44:50.790457 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:50.790459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:50.790839 (+ 380us) server_negotiation.cc:407] Connection header received
0928 04:44:50.791518 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:50.791520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:50.791537 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:50.791611 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:44:50.792599 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:50.793809 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:50.795526 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:50.795857 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:50.795909 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:50.798695 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:44:50.798697 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:44:50.798698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:44:50.798706 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:44:50.800541 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:44:50.801104 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:44:50.801106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:44:50.801106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:44:50.801168 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:44:50.801506 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:44:50.801507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:44:50.801507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:44:50.801674 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:44:50.801719 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:50.802129 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 04:44:50.802238 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 04:44:50.802887 13489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:50.790347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38995 (local address 127.1.226.194:44291)
0928 04:44:50.790550 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 04:44:50.790603 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 04:44:50.791390 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:44:50.791793 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:44:50.791798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:44:50.792066 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:44:50.792448 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:50.792455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:50.794082 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:50.794084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:50.795416 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:50.795418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:50.796006 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:50.796007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:50.796178 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:50.796850 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:44:50.796862 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:44:50.798551 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:44:50.800651 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:44:50.800673 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:44:50.800676 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:44:50.801004 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:44:50.801241 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:44:50.801242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:44:50.801242 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:44:50.801415 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:44:50.801773 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:44:50.801780 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:44:50.801968 (+ 188us) client_negotiation.cc:769] Sending connection context
0928 04:44:50.802444 (+ 476us) client_negotiation.cc:240] Negotiation successful
0928 04:44:50.802626 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":92,"threads_started":1}
I20260928 04:44:50.803561 13487 heartbeater.cc:347] Connected to a master server at 127.1.226.254:38995
I20260928 04:44:50.803675 13487 heartbeater.cc:464] Registering TS with master...
I20260928 04:44:50.803892 13487 heartbeater.cc:511] Master 127.1.226.254:38995 requested a full tablet report, sending...
I20260928 04:44:50.804303 13137 ts_manager.cc:194] Registered new tserver with Master: 16e324bdac7e4c148f496ba72e71480a (127.1.226.194:40043)
I20260928 04:44:50.804813 13137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.194@KRBTEST.COM'} at 127.1.226.194:44291
WARNING: no policy specified for kudu/127.1.226.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
I20260928 04:44:50.827236 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.226.212:47077/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:0
--local_ip_for_outbound_sockets=127.1.226.195
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:38995
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:32781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:44:50.947968 13492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:44:50.948298 13492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:44:50.948406 13492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:44:50.950332 13492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:44:50.950409 13492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:44:50.950453 13492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:44:50.952934 13492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:32781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.226.212:47077/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:38995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:44:50.953389 13492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:44:50.953682 13492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:44:50.959753 13497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:44:50.959762 13500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:44:50.959762 13498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:44:50.960139 13492 server_base.cc:1034] running on GCE node
I20260928 04:44:50.960374 13492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:44:50.960650 13492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:44:50.961809 13492 hybrid_clock.cc:648] HybridClock initialized: now 1790570690961797 us; error 29 us; skew 500 ppm
Sep 28 04:44:50 dist-test-slave-j5c3 krb5kdc[12661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570690, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:44:50.964413 13492 init.cc:377] Logged in from keytab as kudu/127.1.226.195@KRBTEST.COM (short username kudu)
I20260928 04:44:50.964964 13492 webserver.cc:460] Webserver started at http://127.1.226.195:41611/ using document root <none> and password file <none>
I20260928 04:44:50.965227 13492 fs_manager.cc:359] Metadata directory not provided
I20260928 04:44:50.965272 13492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:44:50.965411 13492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 04:44:50 dist-test-slave-j5c3 krb5kdc[12661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570690, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for HTTP/127.1.226.212@KRBTEST.COM
I20260928 04:44:50.980849 13492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-2/data/instance:
uuid: "a65f6065512641788585a25fe8af99d0"
format_stamp: "Formatted at 2026-09-28 04:44:50 on dist-test-slave-j5c3"
server_key: "51cf51551f9ddcf0659d2dc483c84c79"
server_key_iv: "834e94f974eb32b31358a39e6d809f0f"
server_key_version: "kuduclusterkey@0"
I20260928 04:44:50.981446 13492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance:
uuid: "a65f6065512641788585a25fe8af99d0"
format_stamp: "Formatted at 2026-09-28 04:44:50 on dist-test-slave-j5c3"
server_key: "51cf51551f9ddcf0659d2dc483c84c79"
server_key_iv: "834e94f974eb32b31358a39e6d809f0f"
server_key_version: "kuduclusterkey@0"
I20260928 04:44:50.984401 13492 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260928 04:44:51.000659 13507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:51.001011 13492 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 04:44:51.001199 13492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "a65f6065512641788585a25fe8af99d0"
format_stamp: "Formatted at 2026-09-28 04:44:50 on dist-test-slave-j5c3"
server_key: "51cf51551f9ddcf0659d2dc483c84c79"
server_key_iv: "834e94f974eb32b31358a39e6d809f0f"
server_key_version: "kuduclusterkey@0"
I20260928 04:44:51.001376 13492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:44:51.041100 13492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:44:51.044005 13492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:44:51.044263 13492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:44:51.044587 13492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:44:51.045017 13492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:44:51.045105 13492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:51.045288 13492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:44:51.045349 13492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:44:51.052855 13492 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:39423
I20260928 04:44:51.053120 13620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:39423 every 8 connection(s)
I20260928 04:44:51.053470 13492 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
I20260928 04:44:51.054742 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 13492
I20260928 04:44:51.054865 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance
I20260928 04:44:51.055076 1931 external_mini_cluster.cc:1428] Setting key 7be57b7f35b7f6da4fb707eea9e26653
Sep 28 04:44:51 dist-test-slave-j5c3 krb5kdc[12661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570690, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:44:51.066978 13352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:51.055264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:46617 (local address 127.1.226.254:38995)
0928 04:44:51.055340 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:51.055342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:51.055574 (+ 232us) server_negotiation.cc:407] Connection header received
0928 04:44:51.056434 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:51.056435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:51.056447 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:51.056477 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:44:51.057336 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:51.058167 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:51.059715 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:51.060041 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:51.060127 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:51.063133 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:44:51.063135 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:44:51.063136 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:44:51.063144 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:44:51.065134 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:44:51.065691 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:44:51.065692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:44:51.065693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:44:51.065756 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:44:51.066081 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:44:51.066082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:44:51.066083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:44:51.066257 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:44:51.066308 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:51.066758 (+ 450us) server_negotiation.cc:299] Negotiation successful
0928 04:44:51.066858 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 04:44:51.067588 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:51.054312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38995 (local address 127.1.226.195:46617)
0928 04:44:51.055439 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0928 04:44:51.055491 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 04:44:51.056309 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:44:51.056670 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:44:51.056675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:44:51.056947 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:44:51.057224 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:51.057230 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:51.058307 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:51.058308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:51.059625 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:51.059628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:51.060238 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:51.060240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:51.060454 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:51.061231 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:44:51.061242 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:44:51.063020 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:44:51.065245 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:44:51.065252 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:44:51.065255 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:44:51.065580 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:44:51.065837 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:44:51.065838 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:44:51.065838 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:44:51.065993 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:44:51.066364 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:44:51.066372 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:44:51.066556 (+ 184us) client_negotiation.cc:769] Sending connection context
0928 04:44:51.067135 (+ 579us) client_negotiation.cc:240] Negotiation successful
0928 04:44:51.067328 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1008,"thread_start_us":84,"threads_started":1}
I20260928 04:44:51.068116 13621 heartbeater.cc:347] Connected to a master server at 127.1.226.254:38995
I20260928 04:44:51.068260 13621 heartbeater.cc:464] Registering TS with master...
I20260928 04:44:51.068459 13621 heartbeater.cc:511] Master 127.1.226.254:38995 requested a full tablet report, sending...
I20260928 04:44:51.069065 13137 ts_manager.cc:194] Registered new tserver with Master: a65f6065512641788585a25fe8af99d0 (127.1.226.195:39423)
I20260928 04:44:51.069624 13137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.195@KRBTEST.COM'} at 127.1.226.195:46617
I20260928 04:44:51.077750 1931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:44:51.088593 13352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:51.079452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42134 (local address 127.1.226.254:38995)
0928 04:44:51.079501 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:51.079504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:51.079700 (+ 196us) server_negotiation.cc:407] Connection header received
0928 04:44:51.079767 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:51.079768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:51.079780 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:51.079849 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:44:51.080751 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:51.081613 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:51.082916 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:51.083200 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:51.083285 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:51.085061 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:44:51.085062 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:44:51.085063 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:44:51.085070 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:44:51.087039 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:44:51.087483 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:44:51.087485 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:44:51.087485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:44:51.087558 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:44:51.087825 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:44:51.087826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:44:51.087826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:44:51.087971 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:44:51.088249 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:51.088412 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 04:44:51.088499 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:44:51.090898 13137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42134:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:44:51.093474 13137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:44:51.105600 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:51.099116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:45803 (local address 127.0.0.1:55978)
0928 04:44:51.099903 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0928 04:44:51.099946 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 04:44:51.100008 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:44:51.100897 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:44:51.100899 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:44:51.100901 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:44:51.100950 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:51.100951 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:51.102640 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:51.102641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:51.104325 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:51.104328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:51.105307 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:51.105308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:51.105390 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:51.105408 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 04:44:51.105460 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 04:44:51.105511 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":38,"thread_start_us":86,"threads_started":1}
I20260928 04:44:51.106340 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:51.099227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55978 (local address 127.1.226.193:45803)
0928 04:44:51.100718 (+ 1491us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:51.100722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:51.100732 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:44:51.100764 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:51.100766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:51.100783 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:51.101046 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:44:51.101109 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:51.102536 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:51.104423 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:51.105200 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:51.106111 (+ 911us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:51.106185 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:51.106212 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 04:44:51.106256 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1459,"thread_start_us":78,"threads_started":1}
I20260928 04:44:51.107594 13283 tablet_service.cc:1467] Processing CreateTablet for tablet 0ab3716c02304649b84768a7cdda16da (DEFAULT_TABLE table=test-table [id=89a4cfcfaf984e0eb684b6ec24c1aeec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:44:51.107992 13283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ab3716c02304649b84768a7cdda16da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:44:51.109771 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:51.098823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:40043 (local address 127.0.0.1:59084)
0928 04:44:51.099890 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0928 04:44:51.099938 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 04:44:51.100009 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:44:51.100799 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:44:51.100802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:44:51.100805 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:44:51.100882 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:51.100884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:51.102529 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:51.102530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:51.105138 (+ 2608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:51.105141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:51.109504 (+ 4363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:51.109504 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:51.109579 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:51.109584 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:44:51.109629 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 04:44:51.109681 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1030,"mutex_wait_us":1132,"thread_start_us":82,"threads_started":1}
I20260928 04:44:51.110289 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:51.099405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59084 (local address 127.1.226.194:40043)
0928 04:44:51.100441 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:51.100446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:51.100456 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:44:51.100497 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:51.100499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:51.100516 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:51.100592 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:44:51.100995 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:51.102309 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:51.106240 (+ 3931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:51.107450 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:51.107531 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:51.110115 (+ 2584us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:51.110151 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 04:44:51.110208 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":997,"thread_start_us":87,"threads_started":1}
I20260928 04:44:51.112025 13639 tablet_bootstrap.cc:492] T 0ab3716c02304649b84768a7cdda16da P 157eafdd618e42ee9886f73244d9d0f4: Bootstrap starting.
I20260928 04:44:51.112090 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:51.099531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:39423 (local address 127.0.0.1:53512)
0928 04:44:51.100261 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0928 04:44:51.100272 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:44:51.100330 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:44:51.102417 (+ 2087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:44:51.102418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:44:51.102420 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:44:51.102462 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:51.102464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:51.108104 (+ 5640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:51.108105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:51.109392 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:51.109395 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:51.111826 (+ 2431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:51.111827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:51.111895 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:51.111899 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:44:51.111957 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 04:44:51.112004 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":70,"threads_started":1}
I20260928 04:44:51.112845 13639 tablet_bootstrap.cc:654] T 0ab3716c02304649b84768a7cdda16da P 157eafdd618e42ee9886f73244d9d0f4: Neither blocks nor log segments found. Creating new log.
I20260928 04:44:51.113181 13639 log.cc:824] T 0ab3716c02304649b84768a7cdda16da P 157eafdd618e42ee9886f73244d9d0f4: Log is configured to *not* fsync() on all Append() calls
I20260928 04:44:51.113164 13420 tablet_service.cc:1467] Processing CreateTablet for tablet 0ab3716c02304649b84768a7cdda16da (DEFAULT_TABLE table=test-table [id=89a4cfcfaf984e0eb684b6ec24c1aeec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:44:51.113528 13420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ab3716c02304649b84768a7cdda16da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:44:51.113966 13639 tablet_bootstrap.cc:492] T 0ab3716c02304649b84768a7cdda16da P 157eafdd618e42ee9886f73244d9d0f4: No bootstrap required, opened a new log
I20260928 04:44:51.114030 13639 ts_tablet_manager.cc:1397] T 0ab3716c02304649b84768a7cdda16da P 157eafdd618e42ee9886f73244d9d0f4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:44:51.117026 13639 raft_consensus.cc:348] T 0ab3716c02304649b84768a7cdda16da P 157eafdd618e42ee9886f73244d9d0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e324bdac7e4c148f496ba72e71480a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40043 } } peers { permanent_uuid: "a65f6065512641788585a25fe8af99d0" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39423 } } peers { permanent_uuid: "157eafdd618e42ee9886f73244d9d0f4" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 45803 } }
I20260928 04:44:51.117239 13639 raft_consensus.cc:374] T 0ab3716c02304649b84768a7cdda16da P 157eafdd618e42ee9886f73244d9d0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:44:51.117291 13639 raft_consensus.cc:729] T 0ab3716c02304649b84768a7cdda16da P 157eafdd618e42ee9886f73244d9d0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157eafdd618e42ee9886f73244d9d0f4, State: Initialized, Role: FOLLOWER
I20260928 04:44:51.117458 13639 consensus_queue.cc:260] T 0ab3716c02304649b84768a7cdda16da P 157eafdd618e42ee9886f73244d9d0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e324bdac7e4c148f496ba72e71480a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40043 } } peers { permanent_uuid: "a65f6065512641788585a25fe8af99d0" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39423 } } peers { permanent_uuid: "157eafdd618e42ee9886f73244d9d0f4" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 45803 } }
I20260928 04:44:51.117761 13639 ts_tablet_manager.cc:1428] T 0ab3716c02304649b84768a7cdda16da P 157eafdd618e42ee9886f73244d9d0f4: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 04:44:51.117897 13349 heartbeater.cc:503] Master 127.1.226.254:38995 was elected leader, sending a full tablet report...
I20260928 04:44:51.119565 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:51.099693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53512 (local address 127.1.226.195:39423)
0928 04:44:51.100196 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:51.100202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:51.100375 (+ 173us) server_negotiation.cc:407] Connection header received
0928 04:44:51.100433 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:51.100435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:51.100454 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:51.102804 (+ 2350us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:44:51.102895 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:51.107994 (+ 5099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:51.110746 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:51.111603 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:51.111709 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:51.111789 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:51.119399 (+ 7610us) server_negotiation.cc:299] Negotiation successful
0928 04:44:51.119467 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":86,"threads_started":1}
I20260928 04:44:51.120169 13642 tablet_bootstrap.cc:492] T 0ab3716c02304649b84768a7cdda16da P 16e324bdac7e4c148f496ba72e71480a: Bootstrap starting.
I20260928 04:44:51.120534 13549 tablet_service.cc:1467] Processing CreateTablet for tablet 0ab3716c02304649b84768a7cdda16da (DEFAULT_TABLE table=test-table [id=89a4cfcfaf984e0eb684b6ec24c1aeec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:44:51.120930 13549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ab3716c02304649b84768a7cdda16da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:44:51.122164 13642 tablet_bootstrap.cc:654] T 0ab3716c02304649b84768a7cdda16da P 16e324bdac7e4c148f496ba72e71480a: Neither blocks nor log segments found. Creating new log.
I20260928 04:44:51.122606 13642 log.cc:824] T 0ab3716c02304649b84768a7cdda16da P 16e324bdac7e4c148f496ba72e71480a: Log is configured to *not* fsync() on all Append() calls
I20260928 04:44:51.123515 13642 tablet_bootstrap.cc:492] T 0ab3716c02304649b84768a7cdda16da P 16e324bdac7e4c148f496ba72e71480a: No bootstrap required, opened a new log
I20260928 04:44:51.123591 13642 ts_tablet_manager.cc:1397] T 0ab3716c02304649b84768a7cdda16da P 16e324bdac7e4c148f496ba72e71480a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 04:44:51.125610 13642 raft_consensus.cc:348] T 0ab3716c02304649b84768a7cdda16da P 16e324bdac7e4c148f496ba72e71480a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e324bdac7e4c148f496ba72e71480a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40043 } } peers { permanent_uuid: "a65f6065512641788585a25fe8af99d0" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39423 } } peers { permanent_uuid: "157eafdd618e42ee9886f73244d9d0f4" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 45803 } }
I20260928 04:44:51.125797 13642 raft_consensus.cc:374] T 0ab3716c02304649b84768a7cdda16da P 16e324bdac7e4c148f496ba72e71480a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:44:51.125845 13642 raft_consensus.cc:729] T 0ab3716c02304649b84768a7cdda16da P 16e324bdac7e4c148f496ba72e71480a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16e324bdac7e4c148f496ba72e71480a, State: Initialized, Role: FOLLOWER
I20260928 04:44:51.125998 13642 consensus_queue.cc:260] T 0ab3716c02304649b84768a7cdda16da P 16e324bdac7e4c148f496ba72e71480a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e324bdac7e4c148f496ba72e71480a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40043 } } peers { permanent_uuid: "a65f6065512641788585a25fe8af99d0" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39423 } } peers { permanent_uuid: "157eafdd618e42ee9886f73244d9d0f4" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 45803 } }
I20260928 04:44:51.126319 13487 heartbeater.cc:503] Master 127.1.226.254:38995 was elected leader, sending a full tablet report...
I20260928 04:44:51.126928 13642 ts_tablet_manager.cc:1428] T 0ab3716c02304649b84768a7cdda16da P 16e324bdac7e4c148f496ba72e71480a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:44:51.127432 13644 tablet_bootstrap.cc:492] T 0ab3716c02304649b84768a7cdda16da P a65f6065512641788585a25fe8af99d0: Bootstrap starting.
I20260928 04:44:51.128754 13644 tablet_bootstrap.cc:654] T 0ab3716c02304649b84768a7cdda16da P a65f6065512641788585a25fe8af99d0: Neither blocks nor log segments found. Creating new log.
I20260928 04:44:51.129498 13644 log.cc:824] T 0ab3716c02304649b84768a7cdda16da P a65f6065512641788585a25fe8af99d0: Log is configured to *not* fsync() on all Append() calls
I20260928 04:44:51.130302 13644 tablet_bootstrap.cc:492] T 0ab3716c02304649b84768a7cdda16da P a65f6065512641788585a25fe8af99d0: No bootstrap required, opened a new log
I20260928 04:44:51.130393 13644 ts_tablet_manager.cc:1397] T 0ab3716c02304649b84768a7cdda16da P a65f6065512641788585a25fe8af99d0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 04:44:51.131922 13644 raft_consensus.cc:348] T 0ab3716c02304649b84768a7cdda16da P a65f6065512641788585a25fe8af99d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e324bdac7e4c148f496ba72e71480a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40043 } } peers { permanent_uuid: "a65f6065512641788585a25fe8af99d0" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39423 } } peers { permanent_uuid: "157eafdd618e42ee9886f73244d9d0f4" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 45803 } }
I20260928 04:44:51.132084 13644 raft_consensus.cc:374] T 0ab3716c02304649b84768a7cdda16da P a65f6065512641788585a25fe8af99d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:44:51.132130 13644 raft_consensus.cc:729] T 0ab3716c02304649b84768a7cdda16da P a65f6065512641788585a25fe8af99d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a65f6065512641788585a25fe8af99d0, State: Initialized, Role: FOLLOWER
I20260928 04:44:51.132326 13644 consensus_queue.cc:260] T 0ab3716c02304649b84768a7cdda16da P a65f6065512641788585a25fe8af99d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e324bdac7e4c148f496ba72e71480a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40043 } } peers { permanent_uuid: "a65f6065512641788585a25fe8af99d0" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39423 } } peers { permanent_uuid: "157eafdd618e42ee9886f73244d9d0f4" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 45803 } }
I20260928 04:44:51.132683 13621 heartbeater.cc:503] Master 127.1.226.254:38995 was elected leader, sending a full tablet report...
I20260928 04:44:51.132663 13644 ts_tablet_manager.cc:1428] T 0ab3716c02304649b84768a7cdda16da P a65f6065512641788585a25fe8af99d0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 04:44:51.148303 13641 raft_consensus.cc:482] T 0ab3716c02304649b84768a7cdda16da P 157eafdd618e42ee9886f73244d9d0f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:44:51.148469 13641 raft_consensus.cc:504] T 0ab3716c02304649b84768a7cdda16da P 157eafdd618e42ee9886f73244d9d0f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e324bdac7e4c148f496ba72e71480a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40043 } } peers { permanent_uuid: "a65f6065512641788585a25fe8af99d0" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39423 } } peers { permanent_uuid: "157eafdd618e42ee9886f73244d9d0f4" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 45803 } }
I20260928 04:44:51.148895 13641 leader_election.cc:288] T 0ab3716c02304649b84768a7cdda16da P 157eafdd618e42ee9886f73244d9d0f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16e324bdac7e4c148f496ba72e71480a (127.1.226.194:40043), a65f6065512641788585a25fe8af99d0 (127.1.226.195:39423)
I20260928 04:44:51.157948 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:51.149917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:57571 (local address 127.1.226.195:39423)
0928 04:44:51.150007 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:51.150010 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:51.150021 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:44:51.150123 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:51.150133 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:51.152100 (+ 1967us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:51.153428 (+ 1328us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:44:51.154087 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:51.154955 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:51.156837 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:51.157460 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:51.157519 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:51.157529 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:51.157795 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 04:44:51.157858 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 04:44:51.158293 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:51.149146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:39423 (local address 127.1.226.193:57571)
0928 04:44:51.149347 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 04:44:51.149380 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 04:44:51.149707 (+ 327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:44:51.153863 (+ 4156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:44:51.153864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:44:51.153867 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:44:51.153971 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:51.153973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:51.155193 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:51.155194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:51.156715 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:51.156719 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:51.157599 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:51.157600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:51.157703 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:51.157707 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:44:51.158135 (+ 428us) client_negotiation.cc:240] Negotiation successful
0928 04:44:51.158189 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":114,"threads_started":1}
I20260928 04:44:51.158717 13562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ab3716c02304649b84768a7cdda16da" candidate_uuid: "157eafdd618e42ee9886f73244d9d0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a65f6065512641788585a25fe8af99d0" is_pre_election: true
I20260928 04:44:51.158906 13562 raft_consensus.cc:2454] T 0ab3716c02304649b84768a7cdda16da P a65f6065512641788585a25fe8af99d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 157eafdd618e42ee9886f73244d9d0f4 in term 0.
I20260928 04:44:51.159178 13235 leader_election.cc:302] T 0ab3716c02304649b84768a7cdda16da P 157eafdd618e42ee9886f73244d9d0f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157eafdd618e42ee9886f73244d9d0f4, a65f6065512641788585a25fe8af99d0; no voters:
I20260928 04:44:51.159336 13641 raft_consensus.cc:2781] T 0ab3716c02304649b84768a7cdda16da P 157eafdd618e42ee9886f73244d9d0f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:44:51.159394 13641 raft_consensus.cc:482] T 0ab3716c02304649b84768a7cdda16da P 157eafdd618e42ee9886f73244d9d0f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:44:51.159427 13641 raft_consensus.cc:3037] T 0ab3716c02304649b84768a7cdda16da P 157eafdd618e42ee9886f73244d9d0f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:44:51.160645 13641 raft_consensus.cc:504] T 0ab3716c02304649b84768a7cdda16da P 157eafdd618e42ee9886f73244d9d0f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e324bdac7e4c148f496ba72e71480a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40043 } } peers { permanent_uuid: "a65f6065512641788585a25fe8af99d0" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39423 } } peers { permanent_uuid: "157eafdd618e42ee9886f73244d9d0f4" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 45803 } }
I20260928 04:44:51.160827 13641 leader_election.cc:288] T 0ab3716c02304649b84768a7cdda16da P 157eafdd618e42ee9886f73244d9d0f4 [CANDIDATE]: Term 1 election: Requested vote from peers 16e324bdac7e4c148f496ba72e71480a (127.1.226.194:40043), a65f6065512641788585a25fe8af99d0 (127.1.226.195:39423)
I20260928 04:44:51.161073 13562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ab3716c02304649b84768a7cdda16da" candidate_uuid: "157eafdd618e42ee9886f73244d9d0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a65f6065512641788585a25fe8af99d0"
I20260928 04:44:51.161164 13562 raft_consensus.cc:3037] T 0ab3716c02304649b84768a7cdda16da P a65f6065512641788585a25fe8af99d0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:44:51.162163 13562 raft_consensus.cc:2454] T 0ab3716c02304649b84768a7cdda16da P a65f6065512641788585a25fe8af99d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 157eafdd618e42ee9886f73244d9d0f4 in term 1.
I20260928 04:44:51.162366 13235 leader_election.cc:302] T 0ab3716c02304649b84768a7cdda16da P 157eafdd618e42ee9886f73244d9d0f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157eafdd618e42ee9886f73244d9d0f4, a65f6065512641788585a25fe8af99d0; no voters:
I20260928 04:44:51.162461 13641 raft_consensus.cc:2781] T 0ab3716c02304649b84768a7cdda16da P 157eafdd618e42ee9886f73244d9d0f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:44:51.164338 13641 raft_consensus.cc:686] T 0ab3716c02304649b84768a7cdda16da P 157eafdd618e42ee9886f73244d9d0f4 [term 1 LEADER]: Becoming Leader. State: Replica: 157eafdd618e42ee9886f73244d9d0f4, State: Running, Role: LEADER
I20260928 04:44:51.164479 13641 consensus_queue.cc:237] T 0ab3716c02304649b84768a7cdda16da P 157eafdd618e42ee9886f73244d9d0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e324bdac7e4c148f496ba72e71480a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40043 } } peers { permanent_uuid: "a65f6065512641788585a25fe8af99d0" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39423 } } peers { permanent_uuid: "157eafdd618e42ee9886f73244d9d0f4" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 45803 } }
I20260928 04:44:51.166229 13137 catalog_manager.cc:5495] T 0ab3716c02304649b84768a7cdda16da P 157eafdd618e42ee9886f73244d9d0f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 157eafdd618e42ee9886f73244d9d0f4 (127.1.226.193). New cstate: current_term: 1 leader_uuid: "157eafdd618e42ee9886f73244d9d0f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e324bdac7e4c148f496ba72e71480a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a65f6065512641788585a25fe8af99d0" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "157eafdd618e42ee9886f73244d9d0f4" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 45803 } health_report { overall_health: HEALTHY } } }
I20260928 04:44:51.173354 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:51.149609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:57025 (local address 127.1.226.194:40043)
0928 04:44:51.149745 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:51.149748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:51.164359 (+ 14611us) server_negotiation.cc:407] Connection header received
0928 04:44:51.164481 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:51.164483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:51.164496 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:51.164529 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:44:51.164770 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:51.168037 (+ 3267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:51.172254 (+ 4217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:51.172914 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:51.172971 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:51.172980 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:51.173226 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 04:44:51.173272 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260928 04:44:51.173655 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:51.149406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:40043 (local address 127.1.226.193:57025)
0928 04:44:51.164266 (+ 14860us) negotiation.cc:107] Waiting for socket to connect
0928 04:44:51.164281 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 04:44:51.164394 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:44:51.164603 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:44:51.164604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:44:51.164605 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:44:51.164676 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:51.164678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:51.168217 (+ 3539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:51.168218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:51.169565 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:44:51.169567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:44:51.173050 (+ 3483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:44:51.173051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:44:51.173139 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:51.173143 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:44:51.173495 (+ 352us) client_negotiation.cc:240] Negotiation successful
0928 04:44:51.173548 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14833,"thread_start_us":78,"threads_started":1}
I20260928 04:44:51.174130 13440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ab3716c02304649b84768a7cdda16da" candidate_uuid: "157eafdd618e42ee9886f73244d9d0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16e324bdac7e4c148f496ba72e71480a" is_pre_election: true
I20260928 04:44:51.174327 13440 raft_consensus.cc:2454] T 0ab3716c02304649b84768a7cdda16da P 16e324bdac7e4c148f496ba72e71480a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 157eafdd618e42ee9886f73244d9d0f4 in term 0.
I20260928 04:44:51.174467 13439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ab3716c02304649b84768a7cdda16da" candidate_uuid: "157eafdd618e42ee9886f73244d9d0f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16e324bdac7e4c148f496ba72e71480a"
I20260928 04:44:51.174533 13439 raft_consensus.cc:3037] T 0ab3716c02304649b84768a7cdda16da P 16e324bdac7e4c148f496ba72e71480a [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:44:51.175501 13439 raft_consensus.cc:2454] T 0ab3716c02304649b84768a7cdda16da P 16e324bdac7e4c148f496ba72e71480a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 157eafdd618e42ee9886f73244d9d0f4 in term 1.
I20260928 04:44:51.186992 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:44:51.183124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55988 (local address 127.1.226.193:45803)
0928 04:44:51.183193 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 04:44:51.183195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:44:51.183218 (+ 23us) server_negotiation.cc:407] Connection header received
0928 04:44:51.183265 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:44:51.183266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:44:51.183290 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:44:51.183387 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:44:51.183535 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:51.184324 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:51.185597 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:44:51.185916 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:44:51.186005 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:44:51.186543 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:44:51.186711 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:44:51.186820 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 04:44:51.186849 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 04:44:51.186879 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 04:44:51.189453 13562 raft_consensus.cc:1261] T 0ab3716c02304649b84768a7cdda16da P a65f6065512641788585a25fe8af99d0 [term 1 FOLLOWER]: Refusing update from remote peer 157eafdd618e42ee9886f73244d9d0f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:44:51.189858 13650 consensus_queue.cc:1035] T 0ab3716c02304649b84768a7cdda16da P 157eafdd618e42ee9886f73244d9d0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a65f6065512641788585a25fe8af99d0" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 28 04:44:51 dist-test-slave-j5c3 krb5kdc[12661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570690, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for HTTP/127.1.226.212@KRBTEST.COM
I20260928 04:44:51.191833 13439 raft_consensus.cc:1261] T 0ab3716c02304649b84768a7cdda16da P 16e324bdac7e4c148f496ba72e71480a [term 1 FOLLOWER]: Refusing update from remote peer 157eafdd618e42ee9886f73244d9d0f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:44:51.192198 13650 consensus_queue.cc:1035] T 0ab3716c02304649b84768a7cdda16da P 157eafdd618e42ee9886f73244d9d0f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16e324bdac7e4c148f496ba72e71480a" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:44:51.198076 13653 mvcc.cc:205] Tried to move back new op lower bound from 7334177551109148672 to 7334177551013687296. Current Snapshot: MvccSnapshot[applied={T|T < 7334177551109148672}]
I20260928 04:44:51.200343 13654 mvcc.cc:205] Tried to move back new op lower bound from 7334177551109148672 to 7334177551013687296. Current Snapshot: MvccSnapshot[applied={T|T < 7334177551109148672}]
I20260928 04:44:51.202904 13136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:44:51.203019 13136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:44:51.203995 13136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0d7d50620384a0b8c267c226c7f4e17: Sending DeleteTablet for 3 replicas of tablet 0ab3716c02304649b84768a7cdda16da
I20260928 04:44:51.204533 13420 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ab3716c02304649b84768a7cdda16da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:44:51 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:59084
I20260928 04:44:51.204774 13283 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ab3716c02304649b84768a7cdda16da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:44:51 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:55978
I20260928 04:44:51.205192 13549 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ab3716c02304649b84768a7cdda16da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:44:51 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:53512
I20260928 04:44:51.206030 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 13218
I20260928 04:44:51.210286 13661 tablet_replica.cc:330] T 0ab3716c02304649b84768a7cdda16da P 16e324bdac7e4c148f496ba72e71480a: stopping tablet replica
I20260928 04:44:51.210436 13661 raft_consensus.cc:2229] T 0ab3716c02304649b84768a7cdda16da P 16e324bdac7e4c148f496ba72e71480a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:44:51.210507 13661 pending_rounds.cc:62] T 0ab3716c02304649b84768a7cdda16da P 16e324bdac7e4c148f496ba72e71480a: Trying to abort 2 pending ops.
I20260928 04:44:51.210541 13661 pending_rounds.cc:68] T 0ab3716c02304649b84768a7cdda16da P 16e324bdac7e4c148f496ba72e71480a: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7334177551109148672 op_type: WRITE_OP write_request { tablet_id: "0ab3716c02304649b84768a7cdda16da" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\357\343\347\325\006\"8\n\ntest-admin\022*\n 89a4cfcfaf984e0eb684b6ec24c1aeec\020\001\030\001 \001(\001" signature: "<redacted>""\252T\200S\304\025\231\362F\314;\272\243\342K\031\005\212?\367;\234\000\004\003g\250Io\245:\301Pv\216\204g\034j\304>b\375XZQ\254%\351\271+n\256\'\347\225\344\n\027\234\332\221&\306" signing_key_seq_num: 0 } } request_id { client_id: "6282b3e92dc34157a9ae0d3bf142d086" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260928 04:44:51.210803 13661 pending_rounds.cc:68] T 0ab3716c02304649b84768a7cdda16da P 16e324bdac7e4c148f496ba72e71480a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7334177551013687296 op_type: NO_OP noop_request { }
I20260928 04:44:51.210850 13661 raft_consensus.cc:2866] T 0ab3716c02304649b84768a7cdda16da P 16e324bdac7e4c148f496ba72e71480a [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260928 04:44:51.210877 13661 raft_consensus.cc:2258] T 0ab3716c02304649b84768a7cdda16da P 16e324bdac7e4c148f496ba72e71480a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:44:51.211069 13663 tablet_replica.cc:330] T 0ab3716c02304649b84768a7cdda16da P a65f6065512641788585a25fe8af99d0: stopping tablet replica
I20260928 04:44:51.211169 13663 raft_consensus.cc:2229] T 0ab3716c02304649b84768a7cdda16da P a65f6065512641788585a25fe8af99d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:44:51.211282 13663 raft_consensus.cc:2258] T 0ab3716c02304649b84768a7cdda16da P a65f6065512641788585a25fe8af99d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:44:51.211764 13661 ts_tablet_manager.cc:1905] T 0ab3716c02304649b84768a7cdda16da P 16e324bdac7e4c148f496ba72e71480a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:44:51.213886 13661 ts_tablet_manager.cc:1918] T 0ab3716c02304649b84768a7cdda16da P 16e324bdac7e4c148f496ba72e71480a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:44:51.213955 13661 log.cc:1196] T 0ab3716c02304649b84768a7cdda16da P 16e324bdac7e4c148f496ba72e71480a: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-1/wal/wals/0ab3716c02304649b84768a7cdda16da
I20260928 04:44:51.214561 13661 ts_tablet_manager.cc:1939] T 0ab3716c02304649b84768a7cdda16da P 16e324bdac7e4c148f496ba72e71480a: Deleting consensus metadata
I20260928 04:44:51.215225 13124 catalog_manager.cc:4841] TS 16e324bdac7e4c148f496ba72e71480a (127.1.226.194:40043): tablet 0ab3716c02304649b84768a7cdda16da (table test-table [id=89a4cfcfaf984e0eb684b6ec24c1aeec]) successfully deleted
I20260928 04:44:51.224826 13663 ts_tablet_manager.cc:1905] T 0ab3716c02304649b84768a7cdda16da P a65f6065512641788585a25fe8af99d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:44:51.227129 13663 ts_tablet_manager.cc:1918] T 0ab3716c02304649b84768a7cdda16da P a65f6065512641788585a25fe8af99d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:44:51.227322 13663 log.cc:1196] T 0ab3716c02304649b84768a7cdda16da P a65f6065512641788585a25fe8af99d0: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570585693787-1931-0/minicluster-data/ts-2/wal/wals/0ab3716c02304649b84768a7cdda16da
I20260928 04:44:51.228042 13663 ts_tablet_manager.cc:1939] T 0ab3716c02304649b84768a7cdda16da P a65f6065512641788585a25fe8af99d0: Deleting consensus metadata
I20260928 04:44:51.228714 13122 catalog_manager.cc:4841] TS a65f6065512641788585a25fe8af99d0 (127.1.226.195:39423): tablet 0ab3716c02304649b84768a7cdda16da (table test-table [id=89a4cfcfaf984e0eb684b6ec24c1aeec]) successfully deleted
W20260928 04:44:51.231889 13373 connection.cc:535] server connection from 127.1.226.193:57025 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 04:44:51.234843 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.226.193:45803 (error 108)
W20260928 04:44:51.235666 13637 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:44:51.235370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:45803 (local address 127.0.0.1:55994)
0928 04:44:51.235428 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 04:44:51.235439 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:44:51.235482 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:44:51.235613 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.193:45803: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 04:44:51.235956 13121 catalog_manager.cc:4584] TS 157eafdd618e42ee9886f73244d9d0f4 (127.1.226.193:45803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0ab3716c02304649b84768a7cdda16da: Network error: Client connection negotiation failed: client connection to 127.1.226.193:45803: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 04:44:51.236507 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 13355
I20260928 04:44:51.243286 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 13492
I20260928 04:44:51.250342 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 13105
I20260928 04:44:51.258229 1931 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 04:44:51.602982 1931 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 04:44:51.603145 1931 mini_ranger.cc:66] Stopping Ranger...
I20260928 04:44:51.947834 1931 mini_ranger.cc:68] Stopped Ranger
2026-09-28 04:44:51.948 UTC [12691] LOG: received smart shutdown request
2026-09-28 04:44:51.951 UTC [12691] LOG: background worker "logical replication launcher" (PID 12700) exited with exit code 1
2026-09-28 04:44:51.951 UTC [12695] LOG: shutting down
2026-09-28 04:44:52.135 UTC [12691] LOG: database system is shut down
2026-09-28T04:44:52Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47806 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:44:52 dist-test-slave-j5c3 krb5kdc[13673](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:44:52 dist-test-slave-j5c3 krb5kdc[13673](info): set up 2 sockets
Sep 28 04:44:52 dist-test-slave-j5c3 krb5kdc[13673](info): commencing operation
krb5kdc: starting...
W20260928 04:44:53.296200 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:44:53 dist-test-slave-j5c3 krb5kdc[13673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:44:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:44:53Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.226.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.226.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.226.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/rangeradmin_127.1.226.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.226.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/rangeradmin_127.1.226.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.226.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.226.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.226.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/rangerlookup_127.1.226.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.226.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/rangerlookup_127.1.226.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.226.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.226.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.226.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/HTTP_127.1.226.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.226.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/HTTP_127.1.226.212@KRBTEST.COM.keytab.
I20260928 04:44:53.467259 1931 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasky6vTKM/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/postgres -l logfile start
2026-09-28 04:44:56.173 UTC [13703] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 04:44:56.173 UTC [13703] LOG: listening on IPv4 address "127.1.226.212", port 40201
2026-09-28 04:44:56.177 UTC [13703] LOG: listening on Unix socket "/tmp/.s.PGSQL.40201"
2026-09-28 04:44:56.199 UTC [13706] LOG: database system was shut down at 2026-09-28 04:44:54 UTC
2026-09-28 04:44:56.203 UTC [13703] LOG: database system is ready to accept connections
I20260928 04:44:57.168258 1931 mini_postgres.cc:96] Postgres bound to 40201
2026-09-28 04:44:57.174 UTC [13714] FATAL: database "slave" does not exist
127.1.226.212:40201 - accepting connections
I20260928 04:44:57.174875 1931 mini_ranger.cc:161] Starting Ranger...
I20260928 04:44:57.190083 1931 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 04:44:57.698632 1931 mini_ranger.cc:87] Created ranger Postgres database
I20260928 04:44:57.698724 1931 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ranger-admin
2026-09-28 04:44:57,970 [I] DB FLAVOR :POSTGRES
2026-09-28 04:44:57,971 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 04:44:57,971 [I] Checking connection..
2026-09-28 04:44:57,971 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:40201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 04:44:58,218 [I] Checking connection passed.
2026-09-28 04:44:58,218 [I] --------- Verifying version history table ---------
2026-09-28 04:44:58,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:40201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 04:44:58,489 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 04:44:58,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:40201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 04:44:58,727 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 04:44:58,727 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 04:44:58,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:40201/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 04:44:58.981 UTC [13778] WARNING: there is no transaction in progress
2026-09-28 04:44:58,988 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 04:44:58,988 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:40201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 04:44:59,250 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 04:44:59,251 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 04:44:59,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:40201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 04:44:59,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:40201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 04:44:59,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:40201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j5c3.c.gcp-upstream.internal','N') ;"
2026-09-28 04:45:00,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:40201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 04:45:00,284 [I] Table x_portal_user does not exist in database ranger
2026-09-28 04:45:00,284 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:40201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 04:45:00,557 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 04:45:00,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:40201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 04:45:00,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:40201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 04:45:01,066 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 04:45:01,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:40201/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 04:45:02.104 UTC [13886] WARNING: there is no transaction in progress
2026-09-28 04:45:02.122 UTC [13886] WARNING: there is no transaction in progress
2026-09-28 04:45:02.140 UTC [13886] WARNING: there is no transaction in progress
2026-09-28 04:45:02.157 UTC [13886] WARNING: there is no transaction in progress
2026-09-28 04:45:02.174 UTC [13886] WARNING: there is no transaction in progress
2026-09-28 04:45:02.195 UTC [13886] WARNING: there is no transaction in progress
2026-09-28 04:45:02.251 UTC [13886] WARNING: there is no transaction in progress
2026-09-28 04:45:02.259 UTC [13886] WARNING: there is no transaction in progress
2026-09-28 04:45:02.275 UTC [13886] WARNING: there is no transaction in progress
2026-09-28 04:45:02.286 UTC [13886] WARNING: there is no transaction in progress
2026-09-28 04:45:02.297 UTC [13886] WARNING: there is no transaction in progress
2026-09-28 04:45:02.312 UTC [13886] WARNING: there is no transaction in progress
2026-09-28 04:45:02.323 UTC [13886] WARNING: there is no transaction in progress
2026-09-28 04:45:02.330 UTC [13886] WARNING: there is no transaction in progress
2026-09-28 04:45:02.337 UTC [13886] WARNING: there is no transaction in progress
2026-09-28 04:45:02.344 UTC [13886] WARNING: there is no transaction in progress
2026-09-28 04:45:03,039 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 04:45:03,040 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:40201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 04:45:03,286 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 04:45:03,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:40201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 04:45:03,546 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 04:45:03,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:40201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 04:45:03,793 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 04:45:03,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:40201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 04:45:04,054 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:40201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 04:45:04,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:40201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j5c3.c.gcp-upstream.internal';"
2026-09-28 04:45:04,549 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 04:45:04,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:40201/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 04:45:04,789 [I] DB_PATCHES have already been applied
I20260928 04:45:04.795447 1931 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ranger-admin:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasky6vTKM/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 04:45:04.795528 1931 mini_ranger.cc:193] Using host: 127.1.226.212
I20260928 04:45:04.798800 1931 mini_ranger.cc:227] Ranger admin URL: http://127.1.226.212:40759
Sep 28, 2026 4:45:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:45:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:45:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 4:45:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 4:45:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 4:45:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 4:45:05 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 04:45:05.978401 1931 mini_ranger.cc:160] Time spent starting Ranger: real 8.804s user 0.000s sys 0.009s
Sep 28, 2026 4:45:25 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 4:45:25 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 4:45:25 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 4:45:25 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 4:45:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 4:45:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 4:45:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 4:45:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 4:45:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 4:45:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 4:45:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 4:45:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 4:45:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 4:45:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 4:45:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 4:45:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 4:45:25 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 4:45:26 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 04:45:27.300969 1931 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.226.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.226.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.226.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/rangerkms_127.1.226.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.226.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/rangerkms_127.1.226.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.226.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.226.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.226.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/HTTP_127.1.226.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.226.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/HTTP_127.1.226.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 04:45:27 dist-test-slave-j5c3 krb5kdc[13673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570727, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 04:45:27.389736 1931 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 04:45:27.399314 1931 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 04:45:28.272876 1931 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 04:45:28.272969 1931 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ranger-kms
I20260928 04:45:28.272991 1931 mini_ranger_kms.cc:227] Using postgres at 127.1.226.212:40201
2026-09-28 04:45:28,435 [I] DB FLAVOR :POSTGRES
2026-09-28 04:45:28,435 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 04:45:28,435 [I] Checking connection
2026-09-28 04:45:28,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:40201/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 04:45:28,672 [I] connection success
2026-09-28 04:45:28,672 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 04:45:28,672 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 04:45:28,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:40201/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 04:45:28,914 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 04:45:28,915 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 04:45:28,915 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 04:45:28,915 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.226.212:40201/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 04:45:29,214 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 04:45:29.696448 1931 mini_ranger_kms.cc:310] Created kms service
I20260928 04:45:29.890635 1931 mini_ranger_kms.cc:326] Created kudu user
I20260928 04:45:29.934621 1931 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 04:45:29 dist-test-slave-j5c3 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570727, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.226.212@KRBTEST.COM
I20260928 04:45:30.350208 1931 mini_ranger_kms.cc:384] Added ranger policy
I20260928 04:45:30.350401 1931 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ranger-kms:/tmp/dist-test-tasky6vTKM/build/release/bin/postgresql.jar:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasky6vTKM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 04:45:30.350430 1931 mini_ranger_kms.cc:240] Using host: 127.1.226.212
I20260928 04:45:30.353677 1931 mini_ranger_kms.cc:276] Ranger KMS PID: 14066
I20260928 04:45:30.353755 1931 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.226.212:57339
Sep 28, 2026 4:45:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:45:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:45:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 4:45:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 4:45:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 4:45:32 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 04:45:32.805490 1931 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.416s user 0.003s sys 0.143s
I20260928 04:45:32.805599 1931 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 04:45:32.805649 1931 mini_ranger_kms.cc:401] 127.1.226.212:57339/kms/v1/keys
Sep 28 04:45:33 dist-test-slave-j5c3 krb5kdc[13673](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790570733, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.226.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 04:45:33 dist-test-slave-j5c3 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570733, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.226.212@KRBTEST.COM for HTTP/127.1.226.212@KRBTEST.COM
Sep 28 04:45:34 dist-test-slave-j5c3 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570733, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.226.212@KRBTEST.COM for HTTP/127.1.226.212@KRBTEST.COM
Sep 28 04:45:36 dist-test-slave-j5c3 krb5kdc[13673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:45:36.299468 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:39383
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:39383
--ranger_config_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.226.212:57339/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:45:36.391384 14117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:45:36.391618 14117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:45:36.391644 14117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:45:36.393031 14117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:45:36.393082 14117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:45:36.393098 14117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:45:36.393112 14117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:45:36.393126 14117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:45:36.394985 14117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.226.212:57339/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:39383
--ranger_config_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:39383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:45:36.395267 14117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:45:36.395538 14117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:45:36.401108 14125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:36.401005 14122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:36.401264 14117 server_base.cc:1034] running on GCE node
W20260928 04:45:36.401131 14123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:36.401666 14117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:45:36.402041 14117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:45:36.403221 14117 hybrid_clock.cc:648] HybridClock initialized: now 1790570736403210 us; error 34 us; skew 500 ppm
Sep 28 04:45:36 dist-test-slave-j5c3 krb5kdc[13673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570736, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:45:36.406167 14117 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:45:36.406709 14117 webserver.cc:460] Webserver started at http://127.1.226.254:39775/ using document root <none> and password file <none>
I20260928 04:45:36.406948 14117 fs_manager.cc:359] Metadata directory not provided
I20260928 04:45:36.407011 14117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:45:36.407173 14117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 04:45:36 dist-test-slave-j5c3 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570736, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for HTTP/127.1.226.212@KRBTEST.COM
I20260928 04:45:36.464277 14117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "7608e21f9fd044cb81935eea2da40bc4"
format_stamp: "Formatted at 2026-09-28 04:45:36 on dist-test-slave-j5c3"
server_key: "40549a416e8b54527c3f748bb59031ef"
server_key_iv: "9bf33e04f7ccc2bc89fb2bdbeefd3884"
server_key_version: "kuduclusterkey@0"
I20260928 04:45:36.464795 14117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "7608e21f9fd044cb81935eea2da40bc4"
format_stamp: "Formatted at 2026-09-28 04:45:36 on dist-test-slave-j5c3"
server_key: "40549a416e8b54527c3f748bb59031ef"
server_key_iv: "9bf33e04f7ccc2bc89fb2bdbeefd3884"
server_key_version: "kuduclusterkey@0"
I20260928 04:45:36.466823 14117 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 04:45:36.482462 14132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:36.482642 14117 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:45:36.482748 14117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "7608e21f9fd044cb81935eea2da40bc4"
format_stamp: "Formatted at 2026-09-28 04:45:36 on dist-test-slave-j5c3"
server_key: "40549a416e8b54527c3f748bb59031ef"
server_key_iv: "9bf33e04f7ccc2bc89fb2bdbeefd3884"
server_key_version: "kuduclusterkey@0"
I20260928 04:45:36.482847 14117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:45:36.501984 14117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:45:36.504110 14117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:45:36.504299 14117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:45:36.509130 14117 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:39383
I20260928 04:45:36.509167 14184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:39383 every 8 connection(s)
I20260928 04:45:36.509558 14117 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:45:36.510246 14185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:45:36.512609 14185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4: Bootstrap starting.
I20260928 04:45:36.513195 14185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4: Neither blocks nor log segments found. Creating new log.
I20260928 04:45:36.513491 14185 log.cc:824] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4: Log is configured to *not* fsync() on all Append() calls
I20260928 04:45:36.514173 14185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4: No bootstrap required, opened a new log
I20260928 04:45:36.516284 14185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7608e21f9fd044cb81935eea2da40bc4" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 39383 } }
I20260928 04:45:36.516399 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 14117
I20260928 04:45:36.516460 14185 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:45:36.516520 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:45:36.516510 14185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7608e21f9fd044cb81935eea2da40bc4, State: Initialized, Role: FOLLOWER
I20260928 04:45:36.516716 1931 external_mini_cluster.cc:1428] Setting key 6a7eb06b44a17e7856155ea19fba1bc5
I20260928 04:45:36.516692 14185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7608e21f9fd044cb81935eea2da40bc4" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 39383 } }
I20260928 04:45:36.516774 14185 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:45:36.516827 14185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:45:36.516877 14185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:45:36.517661 14185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7608e21f9fd044cb81935eea2da40bc4" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 39383 } }
I20260928 04:45:36.517767 14185 leader_election.cc:302] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7608e21f9fd044cb81935eea2da40bc4; no voters:
I20260928 04:45:36.517978 14185 leader_election.cc:288] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:45:36.518035 14190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:45:36.518222 14190 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [term 1 LEADER]: Becoming Leader. State: Replica: 7608e21f9fd044cb81935eea2da40bc4, State: Running, Role: LEADER
I20260928 04:45:36.518260 14185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:45:36.519038 14190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7608e21f9fd044cb81935eea2da40bc4" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 39383 } }
I20260928 04:45:36.519726 14191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7608e21f9fd044cb81935eea2da40bc4. Latest consensus state: current_term: 1 leader_uuid: "7608e21f9fd044cb81935eea2da40bc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7608e21f9fd044cb81935eea2da40bc4" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 39383 } } }
I20260928 04:45:36.519796 14185 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 04:45:36.519912 14192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7608e21f9fd044cb81935eea2da40bc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7608e21f9fd044cb81935eea2da40bc4" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 39383 } } }
I20260928 04:45:36.519944 14192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [sys.catalog]: This master's current role is: LEADER
I20260928 04:45:36.519815 14191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [sys.catalog]: This master's current role is: LEADER
I20260928 04:45:36.520135 14204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:45:36.523015 14204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 04:45:36 dist-test-slave-j5c3 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:45:36.529994 14204 catalog_manager.cc:1347] Generated new cluster ID: de2a7978e3544525b97e0e41e84a5485
I20260928 04:45:36.530046 14204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:45:36.531100 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:36.517238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51980 (local address 127.1.226.254:39383)
0928 04:45:36.517471 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:36.517482 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:36.517519 (+ 37us) server_negotiation.cc:407] Connection header received
0928 04:45:36.517637 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:36.517645 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:36.517706 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:36.517855 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:36.518717 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:36.520209 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:36.521495 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:36.523186 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:36.523702 (+ 516us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:36.526507 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:36.526512 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:36.526512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:36.526525 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:36.529038 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:36.529485 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:36.529489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:36.529490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:36.529628 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:36.529913 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:36.529914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:36.529914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:36.530257 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:36.530795 (+ 538us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:36.530853 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 04:45:36.530949 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":113,"threads_started":1}
I20260928 04:45:36.542591 14204 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:45:36.543335 14204 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:45:36.547796 14204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4: Generated new TSK 0
I20260928 04:45:36.548022 14204 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.226.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
I20260928 04:45:37.212523 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.226.212:57339/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:0
--local_ip_for_outbound_sockets=127.1.226.193
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:39383
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:45:37.325290 14230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:45:37.325591 14230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:45:37.325634 14230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:45:37.327430 14230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:45:37.327494 14230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:45:37.327536 14230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:45:37.329833 14230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.226.212:57339/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:39383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:45:37.330111 14230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:45:37.330443 14230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:45:37.337204 14235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:37.337488 14236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:37.337607 14238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:37.338696 14230 server_base.cc:1034] running on GCE node
I20260928 04:45:37.338953 14230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:45:37.339298 14230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:45:37.340471 14230 hybrid_clock.cc:648] HybridClock initialized: now 1790570737340438 us; error 48 us; skew 500 ppm
Sep 28 04:45:37 dist-test-slave-j5c3 krb5kdc[13673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570737, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:45:37.344014 14230 init.cc:377] Logged in from keytab as kudu/127.1.226.193@KRBTEST.COM (short username kudu)
I20260928 04:45:37.344614 14230 webserver.cc:460] Webserver started at http://127.1.226.193:39783/ using document root <none> and password file <none>
I20260928 04:45:37.344910 14230 fs_manager.cc:359] Metadata directory not provided
I20260928 04:45:37.344982 14230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:45:37.345125 14230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 04:45:37 dist-test-slave-j5c3 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570737, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for HTTP/127.1.226.212@KRBTEST.COM
I20260928 04:45:37.361469 14230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/data/instance:
uuid: "778bdad5832d48f287dedf586b21d9ec"
format_stamp: "Formatted at 2026-09-28 04:45:37 on dist-test-slave-j5c3"
server_key: "68d19464fa020b2d745fb1e993ae8bd4"
server_key_iv: "5bbe656ddb68d1afa0b3c77450433838"
server_key_version: "kuduclusterkey@0"
I20260928 04:45:37.362214 14230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance:
uuid: "778bdad5832d48f287dedf586b21d9ec"
format_stamp: "Formatted at 2026-09-28 04:45:37 on dist-test-slave-j5c3"
server_key: "68d19464fa020b2d745fb1e993ae8bd4"
server_key_iv: "5bbe656ddb68d1afa0b3c77450433838"
server_key_version: "kuduclusterkey@0"
I20260928 04:45:37.372640 14230 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.000s sys 0.002s
I20260928 04:45:37.390014 14247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:37.396576 14230 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20260928 04:45:37.396772 14230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "778bdad5832d48f287dedf586b21d9ec"
format_stamp: "Formatted at 2026-09-28 04:45:37 on dist-test-slave-j5c3"
server_key: "68d19464fa020b2d745fb1e993ae8bd4"
server_key_iv: "5bbe656ddb68d1afa0b3c77450433838"
server_key_version: "kuduclusterkey@0"
I20260928 04:45:37.396977 14230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:45:37.421792 14230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:45:37.424022 14230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:45:37.424243 14230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:45:37.424522 14230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:45:37.424928 14230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:45:37.424981 14230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:37.425048 14230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:45:37.425093 14230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:37.432463 14230 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:45513
I20260928 04:45:37.432893 14230 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
I20260928 04:45:37.433305 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 14230
I20260928 04:45:37.433398 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance
I20260928 04:45:37.433593 1931 external_mini_cluster.cc:1428] Setting key 42fbbe4ed02821075e759bc3b984a1fe
I20260928 04:45:37.433300 14360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:45513 every 8 connection(s)
Sep 28 04:45:37 dist-test-slave-j5c3 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570737, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.226.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.194@KRBTEST.COM" created.
I20260928 04:45:37.448042 14364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:37.434489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:39383 (local address 127.1.226.193:56655)
0928 04:45:37.436316 (+ 1827us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:37.436356 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:37.436831 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:37.438053 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:37.438058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:37.438311 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:45:37.438594 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:37.438600 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:37.439632 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:37.439633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:37.441105 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:37.441110 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:37.441692 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:37.441693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:37.441861 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:37.442553 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:45:37.442565 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:45:37.444284 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:45:37.446165 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:37.446173 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:37.446175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:37.446415 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:37.446727 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:37.446728 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:37.446729 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:37.446913 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:37.447293 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:45:37.447300 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:45:37.447476 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 04:45:37.447595 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 04:45:37.447799 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1739,"thread_start_us":99,"threads_started":1}
I20260928 04:45:37.448467 14363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:37.434500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:56655 (local address 127.1.226.254:39383)
0928 04:45:37.437788 (+ 3288us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:37.437790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:37.437802 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:45:37.437831 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:37.437832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:37.437847 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:37.437950 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:37.438722 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:37.439512 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:37.441215 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:37.441543 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:37.441611 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:37.444420 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:37.444422 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:37.444422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:37.444429 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:37.446071 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:37.446540 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:37.446542 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:37.446542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:37.446626 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:37.447013 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:37.447014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:37.447014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:37.447181 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:37.447259 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:37.448271 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0928 04:45:37.448372 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3251,"thread_start_us":88,"threads_started":1}
I20260928 04:45:37.448928 14361 heartbeater.cc:347] Connected to a master server at 127.1.226.254:39383
I20260928 04:45:37.449090 14361 heartbeater.cc:464] Registering TS with master...
I20260928 04:45:37.449328 14361 heartbeater.cc:511] Master 127.1.226.254:39383 requested a full tablet report, sending...
I20260928 04:45:37.449957 14149 ts_manager.cc:194] Registered new tserver with Master: 778bdad5832d48f287dedf586b21d9ec (127.1.226.193:45513)
I20260928 04:45:37.450858 14149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.193@KRBTEST.COM'} at 127.1.226.193:56655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
I20260928 04:45:37.463595 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.226.212:57339/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:0
--local_ip_for_outbound_sockets=127.1.226.194
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:39383
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 04:45:37 dist-test-slave-j5c3 krb5kdc[13673](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790570737, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 04:45:37.584478 14367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:45:37.584861 14367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:45:37.584946 14367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:45:37.587016 14367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:45:37.587144 14367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:45:37.587252 14367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:45:37.589838 14367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.226.212:57339/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:39383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:45:37.590584 14367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:45:37.590953 14367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:45:37.597479 14377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:37.597821 14374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:37.598021 14375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:37.599303 14367 server_base.cc:1034] running on GCE node
I20260928 04:45:37.599525 14367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:45:37.599797 14367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:45:37.600970 14367 hybrid_clock.cc:648] HybridClock initialized: now 1790570737600932 us; error 51 us; skew 500 ppm
Sep 28 04:45:37 dist-test-slave-j5c3 krb5kdc[13673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570737, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:45:37.604389 14367 init.cc:377] Logged in from keytab as kudu/127.1.226.194@KRBTEST.COM (short username kudu)
I20260928 04:45:37.604933 14367 webserver.cc:460] Webserver started at http://127.1.226.194:34583/ using document root <none> and password file <none>
I20260928 04:45:37.605206 14367 fs_manager.cc:359] Metadata directory not provided
I20260928 04:45:37.605258 14367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:45:37.605383 14367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 04:45:37 dist-test-slave-j5c3 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570737, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for HTTP/127.1.226.212@KRBTEST.COM
I20260928 04:45:37.621774 14367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/data/instance:
uuid: "477f6e49554f4d51b8dea60ee5070cb7"
format_stamp: "Formatted at 2026-09-28 04:45:37 on dist-test-slave-j5c3"
server_key: "be9307df2ddca903622e29eb6cdf986e"
server_key_iv: "5031465c2dcd70c37edd8cbf94c1dcd3"
server_key_version: "kuduclusterkey@0"
I20260928 04:45:37.622200 14367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance:
uuid: "477f6e49554f4d51b8dea60ee5070cb7"
format_stamp: "Formatted at 2026-09-28 04:45:37 on dist-test-slave-j5c3"
server_key: "be9307df2ddca903622e29eb6cdf986e"
server_key_iv: "5031465c2dcd70c37edd8cbf94c1dcd3"
server_key_version: "kuduclusterkey@0"
I20260928 04:45:37.626355 14367 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260928 04:45:37.644332 14384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:37.644591 14367 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260928 04:45:37.644681 14367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "477f6e49554f4d51b8dea60ee5070cb7"
format_stamp: "Formatted at 2026-09-28 04:45:37 on dist-test-slave-j5c3"
server_key: "be9307df2ddca903622e29eb6cdf986e"
server_key_iv: "5031465c2dcd70c37edd8cbf94c1dcd3"
server_key_version: "kuduclusterkey@0"
I20260928 04:45:37.644784 14367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:45:37.657753 14367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:45:37.660555 14367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:45:37.660718 14367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:45:37.660933 14367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:45:37.661351 14367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:45:37.661446 14367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:37.661527 14367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:45:37.661584 14367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:37.670605 14367 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:44607
I20260928 04:45:37.671168 14367 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
I20260928 04:45:37.671942 14498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:44607 every 8 connection(s)
Sep 28 04:45:37 dist-test-slave-j5c3 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570737, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:45:37.681087 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 14367
I20260928 04:45:37.681190 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance
I20260928 04:45:37.681386 1931 external_mini_cluster.cc:1428] Setting key 94b92df507f68329480403c146f5b244
I20260928 04:45:37.683007 14363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:37.672275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:49733 (local address 127.1.226.254:39383)
0928 04:45:37.672333 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:37.672335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:37.672635 (+ 300us) server_negotiation.cc:407] Connection header received
0928 04:45:37.673331 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:37.673333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:37.673349 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:37.673415 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:37.674186 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:37.674981 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:37.676547 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:37.676851 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:37.676927 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:37.679585 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:37.679587 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:37.679587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:37.679595 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:37.681330 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:37.681844 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:37.681845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:37.681846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:37.681907 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:37.682214 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:37.682215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:37.682215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:37.682354 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:37.682397 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:37.682819 (+ 422us) server_negotiation.cc:299] Negotiation successful
0928 04:45:37.682913 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 04:45:37.683749 14501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:37.672268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:39383 (local address 127.1.226.194:49733)
0928 04:45:37.672462 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:37.672514 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:37.673190 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:37.673505 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:37.673510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:37.673781 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:45:37.674068 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:37.674074 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:37.675107 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:37.675109 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:37.676444 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:37.676448 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:37.676967 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:37.676968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:37.677121 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:37.677791 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:45:37.677801 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:45:37.679477 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:45:37.681427 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:37.681434 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:37.681437 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:37.681741 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:37.681981 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:37.681982 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:37.681982 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:37.682132 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:37.682448 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:45:37.682454 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:45:37.682618 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 04:45:37.683329 (+ 711us) client_negotiation.cc:240] Negotiation successful
0928 04:45:37.683499 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":93,"threads_started":1}
I20260928 04:45:37.684525 14499 heartbeater.cc:347] Connected to a master server at 127.1.226.254:39383
I20260928 04:45:37.684629 14499 heartbeater.cc:464] Registering TS with master...
I20260928 04:45:37.684828 14499 heartbeater.cc:511] Master 127.1.226.254:39383 requested a full tablet report, sending...
I20260928 04:45:37.685233 14149 ts_manager.cc:194] Registered new tserver with Master: 477f6e49554f4d51b8dea60ee5070cb7 (127.1.226.194:44607)
I20260928 04:45:37.685715 14149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.194@KRBTEST.COM'} at 127.1.226.194:49733
WARNING: no policy specified for kudu/127.1.226.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
I20260928 04:45:37.718803 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.226.212:57339/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:0
--local_ip_for_outbound_sockets=127.1.226.195
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:39383
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:45:37.840412 14504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:45:37.840667 14504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:45:37.840708 14504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:45:37.842073 14504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:45:37.842137 14504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:45:37.842181 14504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:45:37.844365 14504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.226.212:57339/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:39383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:45:37.844627 14504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:45:37.844897 14504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:45:37.850567 14510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:37.850610 14509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:37.850894 14504 server_base.cc:1034] running on GCE node
W20260928 04:45:37.851416 14512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:37.851672 14504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:45:37.852005 14504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:45:37.853199 14504 hybrid_clock.cc:648] HybridClock initialized: now 1790570737853174 us; error 40 us; skew 500 ppm
Sep 28 04:45:37 dist-test-slave-j5c3 krb5kdc[13673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570737, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:45:37.856436 14504 init.cc:377] Logged in from keytab as kudu/127.1.226.195@KRBTEST.COM (short username kudu)
I20260928 04:45:37.856979 14504 webserver.cc:460] Webserver started at http://127.1.226.195:43597/ using document root <none> and password file <none>
I20260928 04:45:37.857264 14504 fs_manager.cc:359] Metadata directory not provided
I20260928 04:45:37.857337 14504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:45:37.857499 14504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 04:45:37 dist-test-slave-j5c3 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570737, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for HTTP/127.1.226.212@KRBTEST.COM
I20260928 04:45:37.871605 14504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/data/instance:
uuid: "60a64332b3b844e8bce7c51e5322a287"
format_stamp: "Formatted at 2026-09-28 04:45:37 on dist-test-slave-j5c3"
server_key: "de9dc528725c6abf4a1f9fd62297079e"
server_key_iv: "acb76ffc76bb33c15010fc49b9c6a9ca"
server_key_version: "kuduclusterkey@0"
I20260928 04:45:37.872246 14504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance:
uuid: "60a64332b3b844e8bce7c51e5322a287"
format_stamp: "Formatted at 2026-09-28 04:45:37 on dist-test-slave-j5c3"
server_key: "de9dc528725c6abf4a1f9fd62297079e"
server_key_iv: "acb76ffc76bb33c15010fc49b9c6a9ca"
server_key_version: "kuduclusterkey@0"
I20260928 04:45:37.876561 14504 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.001s
I20260928 04:45:37.918124 14519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:37.918503 14504 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.000s
I20260928 04:45:37.918654 14504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "60a64332b3b844e8bce7c51e5322a287"
format_stamp: "Formatted at 2026-09-28 04:45:37 on dist-test-slave-j5c3"
server_key: "de9dc528725c6abf4a1f9fd62297079e"
server_key_iv: "acb76ffc76bb33c15010fc49b9c6a9ca"
server_key_version: "kuduclusterkey@0"
I20260928 04:45:37.918784 14504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:45:37.948077 14504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:45:37.950901 14504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:45:37.951066 14504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:45:37.951375 14504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:45:37.951753 14504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:45:37.951799 14504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:37.951840 14504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:45:37.951864 14504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:37.958391 14504 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:34981
I20260928 04:45:37.958854 14504 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
I20260928 04:45:37.958859 14632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:34981 every 8 connection(s)
I20260928 04:45:37.961345 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 14504
I20260928 04:45:37.961477 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance
I20260928 04:45:37.961745 1931 external_mini_cluster.cc:1428] Setting key f4b7ef02587640956035b5fc08bd2db4
Sep 28 04:45:37 dist-test-slave-j5c3 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570737, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:45:37.972100 14363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:37.960935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:60841 (local address 127.1.226.254:39383)
0928 04:45:37.960986 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:37.960988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:37.961249 (+ 261us) server_negotiation.cc:407] Connection header received
0928 04:45:37.961972 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:37.961974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:37.961990 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:37.962022 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:37.962874 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:37.963659 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:37.965243 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:37.965565 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:37.965615 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:37.968602 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:37.968604 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:37.968605 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:37.968612 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:37.970458 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:37.970962 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:37.970964 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:37.970964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:37.971026 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:37.971332 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:37.971333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:37.971334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:37.971473 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:37.971518 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:37.971915 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 04:45:37.972002 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 04:45:37.972710 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:37.960908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:39383 (local address 127.1.226.195:60841)
0928 04:45:37.961092 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:37.961145 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:37.961837 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:37.962215 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:37.962220 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:37.962482 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:45:37.962760 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:37.962766 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:37.963777 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:37.963778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:37.965135 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:37.965138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:37.965723 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:37.965725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:37.965868 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:37.966760 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:45:37.966771 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:45:37.968485 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:45:37.970561 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:37.970568 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:37.970570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:37.970870 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:37.971097 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:37.971098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:37.971098 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:37.971232 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:37.971566 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:45:37.971573 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:45:37.971746 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 04:45:37.972298 (+ 552us) client_negotiation.cc:240] Negotiation successful
0928 04:45:37.972471 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":101,"threads_started":1}
I20260928 04:45:37.973146 14633 heartbeater.cc:347] Connected to a master server at 127.1.226.254:39383
I20260928 04:45:37.973244 14633 heartbeater.cc:464] Registering TS with master...
I20260928 04:45:37.973443 14633 heartbeater.cc:511] Master 127.1.226.254:39383 requested a full tablet report, sending...
I20260928 04:45:37.973778 14149 ts_manager.cc:194] Registered new tserver with Master: 60a64332b3b844e8bce7c51e5322a287 (127.1.226.195:34981)
I20260928 04:45:37.974275 14149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.195@KRBTEST.COM'} at 127.1.226.195:60841
I20260928 04:45:37.976842 1931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:45:37.977501 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 14230
I20260928 04:45:37.983515 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 14367
I20260928 04:45:37.989167 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 14504
I20260928 04:45:37.996060 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 14117
I20260928 04:45:38.002111 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:39383
--webserver_interface=127.1.226.254
--webserver_port=39775
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:39383
--ranger_config_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.226.212:57339/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.226.212:57339/kms with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:45:38.085378 14636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:45:38.085605 14636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:45:38.085629 14636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:45:38.086862 14636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:45:38.086905 14636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:45:38.086949 14636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:45:38.086962 14636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:45:38.086974 14636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:45:38.088637 14636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.226.212:57339/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:39383
--ranger_config_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:39383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=39775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:45:38.088842 14636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:45:38.089018 14636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:45:38.094359 14641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:38.094396 14636 server_base.cc:1034] running on GCE node
W20260928 04:45:38.094388 14642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:38.094365 14644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:38.094841 14636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:45:38.095119 14636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:45:38.096251 14636 hybrid_clock.cc:648] HybridClock initialized: now 1790570738096240 us; error 49 us; skew 500 ppm
Sep 28 04:45:38 dist-test-slave-j5c3 krb5kdc[13673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570738, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:45:38.098709 14636 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:45:38.099174 14636 webserver.cc:460] Webserver started at http://127.1.226.254:39775/ using document root <none> and password file <none>
I20260928 04:45:38.099426 14636 fs_manager.cc:359] Metadata directory not provided
I20260928 04:45:38.099502 14636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 04:45:38 dist-test-slave-j5c3 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570738, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for HTTP/127.1.226.212@KRBTEST.COM
I20260928 04:45:38.166103 14636 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:45:38.166986 14652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:38.167177 14636 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:45:38.167275 14636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "7608e21f9fd044cb81935eea2da40bc4"
format_stamp: "Formatted at 2026-09-28 04:45:36 on dist-test-slave-j5c3"
server_key: "40549a416e8b54527c3f748bb59031ef"
server_key_iv: "9bf33e04f7ccc2bc89fb2bdbeefd3884"
server_key_version: "kuduclusterkey@0"
I20260928 04:45:38.167820 14636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:45:38.188342 14636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:45:38.190451 14636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:45:38.190622 14636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:45:38.195250 14636 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:39383
I20260928 04:45:38.195298 14704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:39383 every 8 connection(s)
I20260928 04:45:38.195696 14636 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:45:38.197075 14705 sys_catalog.cc:263] Verifying existing consensus state
I20260928 04:45:38.197746 14705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4: Bootstrap starting.
I20260928 04:45:38.198691 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 14636
I20260928 04:45:38.199011 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.226.212:57339/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:45513
--local_ip_for_outbound_sockets=127.1.226.193
--tserver_master_addrs=127.1.226.254:39383
--webserver_port=39783
--webserver_interface=127.1.226.193
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.226.212:57339/kms with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:45:38.206494 14705 log.cc:824] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4: Log is configured to *not* fsync() on all Append() calls
I20260928 04:45:38.208403 14705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:45:38.208808 14705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4: Bootstrap complete.
I20260928 04:45:38.211280 14705 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7608e21f9fd044cb81935eea2da40bc4" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 39383 } }
I20260928 04:45:38.211531 14705 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7608e21f9fd044cb81935eea2da40bc4, State: Initialized, Role: FOLLOWER
I20260928 04:45:38.211710 14705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7608e21f9fd044cb81935eea2da40bc4" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 39383 } }
I20260928 04:45:38.211788 14705 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:45:38.211864 14705 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:45:38.211913 14705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [term 1 FOLLOWER]: Advancing to term 2
I20260928 04:45:38.213186 14705 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7608e21f9fd044cb81935eea2da40bc4" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 39383 } }
I20260928 04:45:38.213294 14705 leader_election.cc:302] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7608e21f9fd044cb81935eea2da40bc4; no voters:
I20260928 04:45:38.213542 14705 leader_election.cc:288] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 04:45:38.213587 14709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 04:45:38.213702 14709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [term 2 LEADER]: Becoming Leader. State: Replica: 7608e21f9fd044cb81935eea2da40bc4, State: Running, Role: LEADER
I20260928 04:45:38.213826 14705 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:45:38.213855 14709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7608e21f9fd044cb81935eea2da40bc4" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 39383 } }
I20260928 04:45:38.214145 14709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7608e21f9fd044cb81935eea2da40bc4. Latest consensus state: current_term: 2 leader_uuid: "7608e21f9fd044cb81935eea2da40bc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7608e21f9fd044cb81935eea2da40bc4" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 39383 } } }
I20260928 04:45:38.214228 14709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [sys.catalog]: This master's current role is: LEADER
I20260928 04:45:38.214150 14710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7608e21f9fd044cb81935eea2da40bc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7608e21f9fd044cb81935eea2da40bc4" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 39383 } } }
I20260928 04:45:38.214401 14710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4 [sys.catalog]: This master's current role is: LEADER
I20260928 04:45:38.214433 14714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:45:38.215023 14714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:45:38.215174 14714 catalog_manager.cc:1259] Loaded cluster ID: de2a7978e3544525b97e0e41e84a5485
I20260928 04:45:38.215215 14714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:45:38.215384 14705 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 04:45:38.219056 14714 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:45:38.219648 14714 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4: Loaded TSK: 0
I20260928 04:45:38.219973 14714 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 04:45:38.312860 14707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:45:38.313169 14707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:45:38.313231 14707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:45:38.315073 14707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:45:38.315150 14707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:45:38.315198 14707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:45:38.317597 14707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.226.212:57339/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:45513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=39783
--tserver_master_addrs=127.1.226.254:39383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:45:38.317863 14707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:45:38.318142 14707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:45:38.326121 14745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:38.326125 14743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:38.326817 14742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:38.327404 14707 server_base.cc:1034] running on GCE node
I20260928 04:45:38.327620 14707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:45:38.327934 14707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:45:38.329108 14707 hybrid_clock.cc:648] HybridClock initialized: now 1790570738329079 us; error 52 us; skew 500 ppm
Sep 28 04:45:38 dist-test-slave-j5c3 krb5kdc[13673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570738, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:45:38.332326 14707 init.cc:377] Logged in from keytab as kudu/127.1.226.193@KRBTEST.COM (short username kudu)
I20260928 04:45:38.332855 14707 webserver.cc:460] Webserver started at http://127.1.226.193:39783/ using document root <none> and password file <none>
I20260928 04:45:38.333108 14707 fs_manager.cc:359] Metadata directory not provided
I20260928 04:45:38.333181 14707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 04:45:38 dist-test-slave-j5c3 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570738, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for HTTP/127.1.226.212@KRBTEST.COM
I20260928 04:45:38.352618 14707 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260928 04:45:38.353418 14753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:38.353621 14707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:45:38.353716 14707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "778bdad5832d48f287dedf586b21d9ec"
format_stamp: "Formatted at 2026-09-28 04:45:37 on dist-test-slave-j5c3"
server_key: "68d19464fa020b2d745fb1e993ae8bd4"
server_key_iv: "5bbe656ddb68d1afa0b3c77450433838"
server_key_version: "kuduclusterkey@0"
I20260928 04:45:38.354950 14707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:45:38.376374 14707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:45:38.378881 14707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:45:38.379042 14707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:45:38.379271 14707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:45:38.379628 14707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:45:38.379679 14707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:38.379711 14707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:45:38.379729 14707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:38.386754 14707 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:45513
I20260928 04:45:38.387234 14707 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
I20260928 04:45:38.389374 14866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:45513 every 8 connection(s)
I20260928 04:45:38.395596 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 14707
I20260928 04:45:38.395931 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.226.212:57339/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:44607
--local_ip_for_outbound_sockets=127.1.226.194
--tserver_master_addrs=127.1.226.254:39383
--webserver_port=34583
--webserver_interface=127.1.226.194
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.226.212:57339/kms with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 04:45:38 dist-test-slave-j5c3 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570738, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:45:38.408417 14869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:38.390040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:39383 (local address 127.1.226.193:33743)
0928 04:45:38.396277 (+ 6237us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:38.396342 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:38.396900 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:38.397512 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:38.397516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:38.397772 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:45:38.398067 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:38.398075 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:38.399767 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:38.399769 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:38.401167 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:38.401169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:38.402210 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:38.402211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:38.402342 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:38.402987 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:45:38.402996 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:45:38.404631 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:45:38.406708 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:38.406714 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:38.406716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:38.406982 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:38.407224 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:38.407224 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:38.407225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:38.407332 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:38.407752 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:45:38.407757 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:45:38.407886 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 04:45:38.407969 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 04:45:38.408109 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6175,"thread_start_us":84,"threads_started":1}
I20260928 04:45:38.409485 14870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:38.390412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:33743 (local address 127.1.226.254:39383)
0928 04:45:38.396317 (+ 5905us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:38.396328 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:38.396480 (+ 152us) server_negotiation.cc:407] Connection header received
0928 04:45:38.397161 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:38.397168 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:38.397229 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:38.397358 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:38.398218 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:38.399609 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:38.401296 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:38.401994 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:38.402099 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:38.404764 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:38.404767 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:38.404768 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:38.404779 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:38.406611 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:38.407064 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:38.407067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:38.407067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:38.407159 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:38.407390 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:38.407391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:38.407391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:38.407649 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:38.407727 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:38.409115 (+ 1388us) server_negotiation.cc:299] Negotiation successful
0928 04:45:38.409251 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5843,"thread_start_us":85,"threads_started":1}
I20260928 04:45:38.410367 14867 heartbeater.cc:347] Connected to a master server at 127.1.226.254:39383
I20260928 04:45:38.410456 14867 heartbeater.cc:464] Registering TS with master...
I20260928 04:45:38.410643 14867 heartbeater.cc:511] Master 127.1.226.254:39383 requested a full tablet report, sending...
W20260928 04:45:38.411465 14867 heartbeater.cc:644] Failed to heartbeat to 127.1.226.254:39383 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 04:45:38.411525 14867 heartbeater.cc:464] Registering TS with master...
I20260928 04:45:38.411576 14867 heartbeater.cc:511] Master 127.1.226.254:39383 requested a full tablet report, sending...
I20260928 04:45:38.411935 14867 heartbeater.cc:464] Registering TS with master...
I20260928 04:45:38.411998 14867 heartbeater.cc:511] Master 127.1.226.254:39383 requested a full tablet report, sending...
W20260928 04:45:38.412322 14867 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 04:45:38.535701 14871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:45:38.535938 14871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:45:38.535965 14871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:45:38.537421 14871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:45:38.537470 14871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:45:38.537503 14871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:45:38.539143 14871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.226.212:57339/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:44607
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=34583
--tserver_master_addrs=127.1.226.254:39383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:45:38.539429 14871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:45:38.539721 14871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:45:38.546535 14877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:38.546536 14879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:38.546551 14876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:38.547663 14871 server_base.cc:1034] running on GCE node
I20260928 04:45:38.547936 14871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:45:38.548309 14871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:45:38.549508 14871 hybrid_clock.cc:648] HybridClock initialized: now 1790570738549477 us; error 48 us; skew 500 ppm
Sep 28 04:45:38 dist-test-slave-j5c3 krb5kdc[13673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570738, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:45:38.553143 14871 init.cc:377] Logged in from keytab as kudu/127.1.226.194@KRBTEST.COM (short username kudu)
I20260928 04:45:38.553759 14871 webserver.cc:460] Webserver started at http://127.1.226.194:34583/ using document root <none> and password file <none>
I20260928 04:45:38.554059 14871 fs_manager.cc:359] Metadata directory not provided
I20260928 04:45:38.554147 14871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 04:45:38 dist-test-slave-j5c3 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570738, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for HTTP/127.1.226.212@KRBTEST.COM
I20260928 04:45:38.569459 14871 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:45:38.570631 14887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:38.570840 14871 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:45:38.570932 14871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "477f6e49554f4d51b8dea60ee5070cb7"
format_stamp: "Formatted at 2026-09-28 04:45:37 on dist-test-slave-j5c3"
server_key: "be9307df2ddca903622e29eb6cdf986e"
server_key_iv: "5031465c2dcd70c37edd8cbf94c1dcd3"
server_key_version: "kuduclusterkey@0"
I20260928 04:45:38.571293 14871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:45:38.591665 14871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:45:38.594241 14871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:45:38.594424 14871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:45:38.594699 14871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:45:38.595142 14871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:45:38.595199 14871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:38.595233 14871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:45:38.595252 14871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:38.602988 14871 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:44607
I20260928 04:45:38.603503 14871 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
I20260928 04:45:38.604516 15000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:44607 every 8 connection(s)
I20260928 04:45:38.608290 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 14871
I20260928 04:45:38.608609 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.226.212:57339/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:34981
--local_ip_for_outbound_sockets=127.1.226.195
--tserver_master_addrs=127.1.226.254:39383
--webserver_port=43597
--webserver_interface=127.1.226.195
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.226.212:57339/kms with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 04:45:38 dist-test-slave-j5c3 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570738, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:45:38.618950 14870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:38.605241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:36627 (local address 127.1.226.254:39383)
0928 04:45:38.605354 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:38.605356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:38.608019 (+ 2663us) server_negotiation.cc:407] Connection header received
0928 04:45:38.608693 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:38.608695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:38.608714 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:38.608768 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:38.609453 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:38.609964 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:38.611491 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:38.611832 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:38.611907 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:38.614905 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:38.614906 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:38.614907 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:38.614918 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:38.617022 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:38.617621 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:38.617622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:38.617623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:38.617694 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:38.618054 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:38.618055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:38.618055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:38.618224 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:38.618308 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:38.618710 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 04:45:38.618816 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 04:45:38.619432 15003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:38.605249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:39383 (local address 127.1.226.194:36627)
0928 04:45:38.607849 (+ 2600us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:38.607906 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:38.608578 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:38.608859 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:38.608864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:38.609110 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:45:38.609317 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:38.609324 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:38.610067 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:38.610068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:38.611398 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:38.611401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:38.611974 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:38.611976 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:38.612191 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:38.612849 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:45:38.612861 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:45:38.614773 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:45:38.617133 (+ 2360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:38.617140 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:38.617143 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:38.617526 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:38.617783 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:38.617784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:38.617784 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:38.617964 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:38.618318 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:45:38.618326 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:45:38.618528 (+ 202us) client_negotiation.cc:769] Sending connection context
0928 04:45:38.618686 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 04:45:38.618877 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2537,"thread_start_us":128,"threads_started":1}
I20260928 04:45:38.620327 15001 heartbeater.cc:347] Connected to a master server at 127.1.226.254:39383
I20260928 04:45:38.620436 15001 heartbeater.cc:464] Registering TS with master...
I20260928 04:45:38.620656 15001 heartbeater.cc:511] Master 127.1.226.254:39383 requested a full tablet report, sending...
W20260928 04:45:38.621548 15001 heartbeater.cc:644] Failed to heartbeat to 127.1.226.254:39383 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 04:45:38.621624 15001 heartbeater.cc:464] Registering TS with master...
I20260928 04:45:38.621702 15001 heartbeater.cc:511] Master 127.1.226.254:39383 requested a full tablet report, sending...
I20260928 04:45:38.622098 15001 heartbeater.cc:464] Registering TS with master...
I20260928 04:45:38.622177 15001 heartbeater.cc:511] Master 127.1.226.254:39383 requested a full tablet report, sending...
W20260928 04:45:38.622570 15001 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 04:45:38.727790 15004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:45:38.728098 15004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:45:38.728142 15004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:45:38.729988 15004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:45:38.730051 15004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:45:38.730093 15004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:45:38.732223 15004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.226.212:57339/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:34981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=43597
--tserver_master_addrs=127.1.226.254:39383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:45:38.732473 15004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:45:38.732735 15004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:45:38.740065 15011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:38.740065 15010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:38.740923 15013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:38.741392 15004 server_base.cc:1034] running on GCE node
I20260928 04:45:38.741653 15004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:45:38.741966 15004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:45:38.743115 15004 hybrid_clock.cc:648] HybridClock initialized: now 1790570738743109 us; error 33 us; skew 500 ppm
Sep 28 04:45:38 dist-test-slave-j5c3 krb5kdc[13673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570738, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:45:38.746398 15004 init.cc:377] Logged in from keytab as kudu/127.1.226.195@KRBTEST.COM (short username kudu)
I20260928 04:45:38.746995 15004 webserver.cc:460] Webserver started at http://127.1.226.195:43597/ using document root <none> and password file <none>
I20260928 04:45:38.747304 15004 fs_manager.cc:359] Metadata directory not provided
I20260928 04:45:38.747458 15004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 04:45:38 dist-test-slave-j5c3 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570738, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for HTTP/127.1.226.212@KRBTEST.COM
I20260928 04:45:38.763499 15004 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:45:38.764247 15021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:38.764472 15004 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:45:38.764580 15004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "60a64332b3b844e8bce7c51e5322a287"
format_stamp: "Formatted at 2026-09-28 04:45:37 on dist-test-slave-j5c3"
server_key: "de9dc528725c6abf4a1f9fd62297079e"
server_key_iv: "acb76ffc76bb33c15010fc49b9c6a9ca"
server_key_version: "kuduclusterkey@0"
I20260928 04:45:38.765069 15004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:45:38.784379 15004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:45:38.787031 15004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:45:38.787201 15004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:45:38.787468 15004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:45:38.787875 15004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:45:38.787925 15004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:38.787955 15004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:45:38.787979 15004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:38.794798 15004 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:34981
I20260928 04:45:38.795292 15004 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
I20260928 04:45:38.797253 15134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:34981 every 8 connection(s)
I20260928 04:45:38.798930 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 15004
Sep 28 04:45:38 dist-test-slave-j5c3 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570738, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:45:38.808907 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:38.797971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:39383 (local address 127.1.226.195:57663)
0928 04:45:38.798137 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:38.798181 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:38.798690 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:38.798964 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:38.798968 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:38.799236 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:45:38.799548 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:38.799554 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:38.800626 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:38.800627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:38.801924 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:38.801928 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:38.802483 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:38.802485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:38.802627 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:38.803277 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:45:38.803288 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:45:38.805059 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:45:38.807087 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:38.807094 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:38.807097 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:38.807419 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:38.807640 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:38.807640 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:38.807641 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:38.807782 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:38.808124 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:45:38.808130 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:45:38.808350 (+ 220us) client_negotiation.cc:769] Sending connection context
0928 04:45:38.808468 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 04:45:38.808633 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260928 04:45:38.809444 14870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:38.797958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:57663 (local address 127.1.226.254:39383)
0928 04:45:38.798011 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:38.798014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:38.798286 (+ 272us) server_negotiation.cc:407] Connection header received
0928 04:45:38.798814 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:38.798815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:38.798839 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:38.798889 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:38.799648 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:38.800503 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:38.802029 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:38.802335 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:38.802396 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:38.805179 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:38.805181 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:38.805181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:38.805189 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:38.806986 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:38.807510 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:38.807511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:38.807511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:38.807573 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:38.807880 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:38.807881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:38.807881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:38.808029 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:38.808095 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:38.809251 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0928 04:45:38.809338 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:45:38.810230 15135 heartbeater.cc:347] Connected to a master server at 127.1.226.254:39383
I20260928 04:45:38.810359 15135 heartbeater.cc:464] Registering TS with master...
I20260928 04:45:38.810551 15135 heartbeater.cc:511] Master 127.1.226.254:39383 requested a full tablet report, sending...
W20260928 04:45:38.811219 15135 heartbeater.cc:644] Failed to heartbeat to 127.1.226.254:39383 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 04:45:38.811305 15135 heartbeater.cc:464] Registering TS with master...
I20260928 04:45:38.811378 15135 heartbeater.cc:511] Master 127.1.226.254:39383 requested a full tablet report, sending...
I20260928 04:45:38.811661 15135 heartbeater.cc:464] Registering TS with master...
I20260928 04:45:38.811736 15135 heartbeater.cc:511] Master 127.1.226.254:39383 requested a full tablet report, sending...
W20260928 04:45:38.812001 15135 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 04:45:38.817373 15139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:38.799361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51988 (local address 127.1.226.254:39383)
0928 04:45:38.799865 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:38.799867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:38.799874 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:45:38.799893 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:38.799894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:38.799904 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:38.799963 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:38.800749 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:38.801502 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:38.812648 (+ 11146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:38.812970 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:38.813062 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:38.814437 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:38.814439 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:38.814439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:38.814447 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:38.816122 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:38.816514 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:38.816515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:38.816515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:38.816581 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:38.816779 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:38.816780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:38.816780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:38.816929 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:38.817117 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:38.817157 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 04:45:38.817260 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":65,"threads_started":1}
Sep 28 04:45:39 dist-test-slave-j5c3 krb5kdc[13673](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790570739, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:45:39.412503 14867 heartbeater.cc:464] Registering TS with master...
I20260928 04:45:39.412667 14867 heartbeater.cc:511] Master 127.1.226.254:39383 requested a full tablet report, sending...
I20260928 04:45:39.413059 14669 ts_manager.cc:194] Registered new tserver with Master: 778bdad5832d48f287dedf586b21d9ec (127.1.226.193:45513)
I20260928 04:45:39.414194 14669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.193@KRBTEST.COM'} at 127.1.226.193:33743
I20260928 04:45:39.622769 15001 heartbeater.cc:464] Registering TS with master...
I20260928 04:45:39.622937 15001 heartbeater.cc:511] Master 127.1.226.254:39383 requested a full tablet report, sending...
I20260928 04:45:39.623423 14669 ts_manager.cc:194] Registered new tserver with Master: 477f6e49554f4d51b8dea60ee5070cb7 (127.1.226.194:44607)
I20260928 04:45:39.623944 14669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.194@KRBTEST.COM'} at 127.1.226.194:36627
Sep 28 04:45:39 dist-test-slave-j5c3 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570739, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for HTTP/127.1.226.212@KRBTEST.COM
I20260928 04:45:39.812201 15135 heartbeater.cc:464] Registering TS with master...
I20260928 04:45:39.812372 15135 heartbeater.cc:511] Master 127.1.226.254:39383 requested a full tablet report, sending...
I20260928 04:45:39.812743 14669 ts_manager.cc:194] Registered new tserver with Master: 60a64332b3b844e8bce7c51e5322a287 (127.1.226.195:34981)
I20260928 04:45:39.813259 14669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.195@KRBTEST.COM'} at 127.1.226.195:57663
I20260928 04:45:39.822269 1931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:45:39.831295 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:39.823300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52004 (local address 127.1.226.254:39383)
0928 04:45:39.823564 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:39.823566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:39.823574 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:45:39.823596 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:39.823597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:39.823609 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:39.823696 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:39.824386 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:39.825189 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:39.826163 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:39.826482 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:39.826562 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:39.827999 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:39.828000 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:39.828001 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:39.828008 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:39.829879 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:39.830280 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:39.830281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:39.830282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:39.830349 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:39.830572 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:39.830574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:39.830574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:39.830735 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:39.830842 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:39.830984 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 04:45:39.831071 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":78,"threads_started":1}
I20260928 04:45:39.833644 14669 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:45:39.836123 14669 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:45:39.845062 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:39.839863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:44607 (local address 127.0.0.1:49494)
0928 04:45:39.840976 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:39.840990 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:39.841049 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:39.841323 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:39.841324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:39.841326 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:45:39.841378 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:39.841381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:39.842591 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:39.842592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:39.843766 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:39.843769 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:39.844774 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:39.844775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:39.844857 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:39.844871 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 04:45:39.844935 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 04:45:39.844984 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1094,"thread_start_us":58,"threads_started":1}
I20260928 04:45:39.845464 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:39.840460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49494 (local address 127.1.226.194:44607)
0928 04:45:39.840912 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:39.840918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:39.841109 (+ 191us) server_negotiation.cc:407] Connection header received
0928 04:45:39.841147 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:39.841150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:39.841167 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:39.841239 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:45:39.841457 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:39.842493 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:39.843862 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:39.844601 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:39.844668 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:39.844726 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:39.845353 (+ 627us) server_negotiation.cc:299] Negotiation successful
0928 04:45:39.845395 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":50,"threads_started":1}
I20260928 04:45:39.845598 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:39.840306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38782 (local address 127.1.226.195:34981)
0928 04:45:39.840619 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:39.840623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:39.840634 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:45:39.840671 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:39.840673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:39.840688 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:39.840786 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:45:39.840999 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:39.842092 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:39.843883 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:39.845011 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:39.845082 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:39.845136 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:39.845486 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 04:45:39.845527 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":68,"threads_started":1}
I20260928 04:45:39.845417 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:39.839630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:34981 (local address 127.0.0.1:38782)
0928 04:45:39.839838 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:39.839883 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:39.839954 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:39.840816 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:39.840819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:39.840822 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:45:39.840896 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:39.840898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:39.842472 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:39.842473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:39.843776 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:39.843779 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:39.845173 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:39.845175 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:39.845240 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:39.845245 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:45:39.845300 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 04:45:39.845342 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":27,"thread_start_us":86,"threads_started":1}
I20260928 04:45:39.846652 15053 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab933b86de34f0283456231b01db9e5 (DEFAULT_TABLE table=test-table [id=29283e6712b24c7f89e005a4d4741f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:45:39.846650 14935 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab933b86de34f0283456231b01db9e5 (DEFAULT_TABLE table=test-table [id=29283e6712b24c7f89e005a4d4741f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:45:39.847026 15053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab933b86de34f0283456231b01db9e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:45:39.847038 14935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab933b86de34f0283456231b01db9e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:45:39.851260 15169 tablet_bootstrap.cc:492] T 9ab933b86de34f0283456231b01db9e5 P 477f6e49554f4d51b8dea60ee5070cb7: Bootstrap starting.
I20260928 04:45:39.852370 15169 tablet_bootstrap.cc:654] T 9ab933b86de34f0283456231b01db9e5 P 477f6e49554f4d51b8dea60ee5070cb7: Neither blocks nor log segments found. Creating new log.
I20260928 04:45:39.852371 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:39.840022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:45513 (local address 127.0.0.1:38322)
0928 04:45:39.845869 (+ 5847us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:39.845880 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:39.845923 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:39.848496 (+ 2573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:39.848497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:39.848499 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:45:39.848535 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:39.848537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:39.849804 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:39.849805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:39.851079 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:39.851082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:39.852125 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:39.852126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:39.852208 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:39.852212 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:45:39.852258 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 04:45:39.852299 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5834,"thread_start_us":64,"threads_started":1}
I20260928 04:45:39.852667 15169 log.cc:824] T 9ab933b86de34f0283456231b01db9e5 P 477f6e49554f4d51b8dea60ee5070cb7: Log is configured to *not* fsync() on all Append() calls
I20260928 04:45:39.852867 15165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:39.840128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38322 (local address 127.1.226.193:45513)
0928 04:45:39.848308 (+ 8180us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:39.848313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:39.848326 (+ 13us) server_negotiation.cc:407] Connection header received
0928 04:45:39.848363 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:39.848365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:39.848382 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:39.848486 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:45:39.848636 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:39.849689 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:39.851169 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:39.851974 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:39.852040 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:39.852093 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:39.852628 (+ 535us) server_negotiation.cc:299] Negotiation successful
0928 04:45:39.852676 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8140,"thread_start_us":108,"threads_started":1}
I20260928 04:45:39.853012 15168 tablet_bootstrap.cc:492] T 9ab933b86de34f0283456231b01db9e5 P 60a64332b3b844e8bce7c51e5322a287: Bootstrap starting.
I20260928 04:45:39.853621 15168 tablet_bootstrap.cc:654] T 9ab933b86de34f0283456231b01db9e5 P 60a64332b3b844e8bce7c51e5322a287: Neither blocks nor log segments found. Creating new log.
I20260928 04:45:39.853749 14801 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab933b86de34f0283456231b01db9e5 (DEFAULT_TABLE table=test-table [id=29283e6712b24c7f89e005a4d4741f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:45:39.853920 15168 log.cc:824] T 9ab933b86de34f0283456231b01db9e5 P 60a64332b3b844e8bce7c51e5322a287: Log is configured to *not* fsync() on all Append() calls
I20260928 04:45:39.854100 14801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab933b86de34f0283456231b01db9e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:45:39.854707 15168 tablet_bootstrap.cc:492] T 9ab933b86de34f0283456231b01db9e5 P 60a64332b3b844e8bce7c51e5322a287: No bootstrap required, opened a new log
I20260928 04:45:39.854833 15168 ts_tablet_manager.cc:1397] T 9ab933b86de34f0283456231b01db9e5 P 60a64332b3b844e8bce7c51e5322a287: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:45:39.854988 15169 tablet_bootstrap.cc:492] T 9ab933b86de34f0283456231b01db9e5 P 477f6e49554f4d51b8dea60ee5070cb7: No bootstrap required, opened a new log
I20260928 04:45:39.855060 15169 ts_tablet_manager.cc:1397] T 9ab933b86de34f0283456231b01db9e5 P 477f6e49554f4d51b8dea60ee5070cb7: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260928 04:45:39.857575 15169 raft_consensus.cc:348] T 9ab933b86de34f0283456231b01db9e5 P 477f6e49554f4d51b8dea60ee5070cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60a64332b3b844e8bce7c51e5322a287" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34981 } } peers { permanent_uuid: "477f6e49554f4d51b8dea60ee5070cb7" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44607 } } peers { permanent_uuid: "778bdad5832d48f287dedf586b21d9ec" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 45513 } }
I20260928 04:45:39.857761 15169 raft_consensus.cc:374] T 9ab933b86de34f0283456231b01db9e5 P 477f6e49554f4d51b8dea60ee5070cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:45:39.857803 15169 raft_consensus.cc:729] T 9ab933b86de34f0283456231b01db9e5 P 477f6e49554f4d51b8dea60ee5070cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 477f6e49554f4d51b8dea60ee5070cb7, State: Initialized, Role: FOLLOWER
I20260928 04:45:39.857661 15168 raft_consensus.cc:348] T 9ab933b86de34f0283456231b01db9e5 P 60a64332b3b844e8bce7c51e5322a287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60a64332b3b844e8bce7c51e5322a287" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34981 } } peers { permanent_uuid: "477f6e49554f4d51b8dea60ee5070cb7" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44607 } } peers { permanent_uuid: "778bdad5832d48f287dedf586b21d9ec" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 45513 } }
I20260928 04:45:39.857934 15169 consensus_queue.cc:260] T 9ab933b86de34f0283456231b01db9e5 P 477f6e49554f4d51b8dea60ee5070cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60a64332b3b844e8bce7c51e5322a287" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34981 } } peers { permanent_uuid: "477f6e49554f4d51b8dea60ee5070cb7" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44607 } } peers { permanent_uuid: "778bdad5832d48f287dedf586b21d9ec" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 45513 } }
I20260928 04:45:39.858068 15168 raft_consensus.cc:374] T 9ab933b86de34f0283456231b01db9e5 P 60a64332b3b844e8bce7c51e5322a287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:45:39.858150 15168 raft_consensus.cc:729] T 9ab933b86de34f0283456231b01db9e5 P 60a64332b3b844e8bce7c51e5322a287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60a64332b3b844e8bce7c51e5322a287, State: Initialized, Role: FOLLOWER
I20260928 04:45:39.858174 15169 ts_tablet_manager.cc:1428] T 9ab933b86de34f0283456231b01db9e5 P 477f6e49554f4d51b8dea60ee5070cb7: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260928 04:45:39.858282 15001 heartbeater.cc:503] Master 127.1.226.254:39383 was elected leader, sending a full tablet report...
I20260928 04:45:39.858368 15168 consensus_queue.cc:260] T 9ab933b86de34f0283456231b01db9e5 P 60a64332b3b844e8bce7c51e5322a287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60a64332b3b844e8bce7c51e5322a287" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34981 } } peers { permanent_uuid: "477f6e49554f4d51b8dea60ee5070cb7" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44607 } } peers { permanent_uuid: "778bdad5832d48f287dedf586b21d9ec" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 45513 } }
I20260928 04:45:39.858656 15168 ts_tablet_manager.cc:1428] T 9ab933b86de34f0283456231b01db9e5 P 60a64332b3b844e8bce7c51e5322a287: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 04:45:39.858803 15135 heartbeater.cc:503] Master 127.1.226.254:39383 was elected leader, sending a full tablet report...
I20260928 04:45:39.864004 15172 tablet_bootstrap.cc:492] T 9ab933b86de34f0283456231b01db9e5 P 778bdad5832d48f287dedf586b21d9ec: Bootstrap starting.
I20260928 04:45:39.864878 15172 tablet_bootstrap.cc:654] T 9ab933b86de34f0283456231b01db9e5 P 778bdad5832d48f287dedf586b21d9ec: Neither blocks nor log segments found. Creating new log.
I20260928 04:45:39.865227 15172 log.cc:824] T 9ab933b86de34f0283456231b01db9e5 P 778bdad5832d48f287dedf586b21d9ec: Log is configured to *not* fsync() on all Append() calls
I20260928 04:45:39.866217 15172 tablet_bootstrap.cc:492] T 9ab933b86de34f0283456231b01db9e5 P 778bdad5832d48f287dedf586b21d9ec: No bootstrap required, opened a new log
I20260928 04:45:39.866307 15172 ts_tablet_manager.cc:1397] T 9ab933b86de34f0283456231b01db9e5 P 778bdad5832d48f287dedf586b21d9ec: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:45:39.869715 15172 raft_consensus.cc:348] T 9ab933b86de34f0283456231b01db9e5 P 778bdad5832d48f287dedf586b21d9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60a64332b3b844e8bce7c51e5322a287" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34981 } } peers { permanent_uuid: "477f6e49554f4d51b8dea60ee5070cb7" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44607 } } peers { permanent_uuid: "778bdad5832d48f287dedf586b21d9ec" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 45513 } }
I20260928 04:45:39.869933 15172 raft_consensus.cc:374] T 9ab933b86de34f0283456231b01db9e5 P 778bdad5832d48f287dedf586b21d9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:45:39.869973 15172 raft_consensus.cc:729] T 9ab933b86de34f0283456231b01db9e5 P 778bdad5832d48f287dedf586b21d9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778bdad5832d48f287dedf586b21d9ec, State: Initialized, Role: FOLLOWER
I20260928 04:45:39.870123 15172 consensus_queue.cc:260] T 9ab933b86de34f0283456231b01db9e5 P 778bdad5832d48f287dedf586b21d9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60a64332b3b844e8bce7c51e5322a287" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34981 } } peers { permanent_uuid: "477f6e49554f4d51b8dea60ee5070cb7" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44607 } } peers { permanent_uuid: "778bdad5832d48f287dedf586b21d9ec" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 45513 } }
I20260928 04:45:39.871053 15172 ts_tablet_manager.cc:1428] T 9ab933b86de34f0283456231b01db9e5 P 778bdad5832d48f287dedf586b21d9ec: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260928 04:45:39.871208 14867 heartbeater.cc:503] Master 127.1.226.254:39383 was elected leader, sending a full tablet report...
W20260928 04:45:39.892835 14868 tablet.cc:2369] T 9ab933b86de34f0283456231b01db9e5 P 778bdad5832d48f287dedf586b21d9ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 04:45:39 dist-test-slave-j5c3 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570739, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for HTTP/127.1.226.212@KRBTEST.COM
I20260928 04:45:39.937708 15176 raft_consensus.cc:482] T 9ab933b86de34f0283456231b01db9e5 P 778bdad5832d48f287dedf586b21d9ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:45:39.937845 15176 raft_consensus.cc:504] T 9ab933b86de34f0283456231b01db9e5 P 778bdad5832d48f287dedf586b21d9ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60a64332b3b844e8bce7c51e5322a287" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34981 } } peers { permanent_uuid: "477f6e49554f4d51b8dea60ee5070cb7" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44607 } } peers { permanent_uuid: "778bdad5832d48f287dedf586b21d9ec" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 45513 } }
I20260928 04:45:39.938283 15176 leader_election.cc:288] T 9ab933b86de34f0283456231b01db9e5 P 778bdad5832d48f287dedf586b21d9ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60a64332b3b844e8bce7c51e5322a287 (127.1.226.195:34981), 477f6e49554f4d51b8dea60ee5070cb7 (127.1.226.194:44607)
I20260928 04:45:39.942282 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:39.938599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:58761 (local address 127.1.226.195:34981)
0928 04:45:39.938875 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:39.938877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:39.938887 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:45:39.938915 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:39.938916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:39.938926 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:39.938999 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:45:39.939227 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:39.939947 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:39.941349 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:39.941892 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:39.941980 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:39.941991 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:39.942163 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 04:45:39.942213 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248}
I20260928 04:45:39.942619 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:39.938522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:34981 (local address 127.1.226.193:58761)
0928 04:45:39.938697 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:39.938709 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:39.938777 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:39.939007 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:39.939007 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:39.939009 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:45:39.939092 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:39.939093 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:39.940049 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:39.940050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:39.941251 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:39.941254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:39.942004 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:39.942005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:39.942070 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:39.942074 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:45:39.942481 (+ 407us) client_negotiation.cc:240] Negotiation successful
0928 04:45:39.942530 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":112,"threads_started":1}
I20260928 04:45:39.942945 15083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab933b86de34f0283456231b01db9e5" candidate_uuid: "778bdad5832d48f287dedf586b21d9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60a64332b3b844e8bce7c51e5322a287" is_pre_election: true
I20260928 04:45:39.943125 15083 raft_consensus.cc:2454] T 9ab933b86de34f0283456231b01db9e5 P 60a64332b3b844e8bce7c51e5322a287 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 778bdad5832d48f287dedf586b21d9ec in term 0.
I20260928 04:45:39.943383 14756 leader_election.cc:302] T 9ab933b86de34f0283456231b01db9e5 P 778bdad5832d48f287dedf586b21d9ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60a64332b3b844e8bce7c51e5322a287, 778bdad5832d48f287dedf586b21d9ec; no voters:
I20260928 04:45:39.943586 15176 raft_consensus.cc:2781] T 9ab933b86de34f0283456231b01db9e5 P 778bdad5832d48f287dedf586b21d9ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:45:39.943646 15176 raft_consensus.cc:482] T 9ab933b86de34f0283456231b01db9e5 P 778bdad5832d48f287dedf586b21d9ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:45:39.943668 15176 raft_consensus.cc:3037] T 9ab933b86de34f0283456231b01db9e5 P 778bdad5832d48f287dedf586b21d9ec [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:45:39.944677 15176 raft_consensus.cc:504] T 9ab933b86de34f0283456231b01db9e5 P 778bdad5832d48f287dedf586b21d9ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60a64332b3b844e8bce7c51e5322a287" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34981 } } peers { permanent_uuid: "477f6e49554f4d51b8dea60ee5070cb7" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44607 } } peers { permanent_uuid: "778bdad5832d48f287dedf586b21d9ec" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 45513 } }
I20260928 04:45:39.944806 15176 leader_election.cc:288] T 9ab933b86de34f0283456231b01db9e5 P 778bdad5832d48f287dedf586b21d9ec [CANDIDATE]: Term 1 election: Requested vote from peers 60a64332b3b844e8bce7c51e5322a287 (127.1.226.195:34981), 477f6e49554f4d51b8dea60ee5070cb7 (127.1.226.194:44607)
I20260928 04:45:39.945117 15083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab933b86de34f0283456231b01db9e5" candidate_uuid: "778bdad5832d48f287dedf586b21d9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60a64332b3b844e8bce7c51e5322a287"
I20260928 04:45:39.945214 15083 raft_consensus.cc:3037] T 9ab933b86de34f0283456231b01db9e5 P 60a64332b3b844e8bce7c51e5322a287 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:45:39.946117 15083 raft_consensus.cc:2454] T 9ab933b86de34f0283456231b01db9e5 P 60a64332b3b844e8bce7c51e5322a287 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 778bdad5832d48f287dedf586b21d9ec in term 1.
I20260928 04:45:39.946318 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:39.939126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:37727 (local address 127.1.226.194:44607)
0928 04:45:39.939178 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:39.939180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:39.942898 (+ 3718us) server_negotiation.cc:407] Connection header received
0928 04:45:39.943003 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:39.943005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:39.943032 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:39.943062 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:45:39.943373 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:39.944053 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:39.945461 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:39.945981 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:39.946026 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:39.946034 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:39.946214 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 04:45:39.946254 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 04:45:39.946352 14756 leader_election.cc:302] T 9ab933b86de34f0283456231b01db9e5 P 778bdad5832d48f287dedf586b21d9ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60a64332b3b844e8bce7c51e5322a287, 778bdad5832d48f287dedf586b21d9ec; no voters:
I20260928 04:45:39.946602 15176 raft_consensus.cc:2781] T 9ab933b86de34f0283456231b01db9e5 P 778bdad5832d48f287dedf586b21d9ec [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:45:39.946708 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:39.938753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:44607 (local address 127.1.226.193:37727)
0928 04:45:39.942844 (+ 4091us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:39.942856 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:39.942927 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:39.943190 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:39.943191 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:39.943192 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:45:39.943223 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:39.943224 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:39.944183 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:39.944183 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:39.945354 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:39.945357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:39.946088 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:39.946088 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:39.946142 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:39.946144 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:45:39.946556 (+ 412us) client_negotiation.cc:240] Negotiation successful
0928 04:45:39.946597 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4079,"thread_start_us":66,"threads_started":1}
I20260928 04:45:39.946784 15176 raft_consensus.cc:686] T 9ab933b86de34f0283456231b01db9e5 P 778bdad5832d48f287dedf586b21d9ec [term 1 LEADER]: Becoming Leader. State: Replica: 778bdad5832d48f287dedf586b21d9ec, State: Running, Role: LEADER
I20260928 04:45:39.946875 15176 consensus_queue.cc:237] T 9ab933b86de34f0283456231b01db9e5 P 778bdad5832d48f287dedf586b21d9ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60a64332b3b844e8bce7c51e5322a287" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34981 } } peers { permanent_uuid: "477f6e49554f4d51b8dea60ee5070cb7" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44607 } } peers { permanent_uuid: "778bdad5832d48f287dedf586b21d9ec" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 45513 } }
I20260928 04:45:39.947431 14955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab933b86de34f0283456231b01db9e5" candidate_uuid: "778bdad5832d48f287dedf586b21d9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "477f6e49554f4d51b8dea60ee5070cb7" is_pre_election: true
I20260928 04:45:39.947575 14955 raft_consensus.cc:2454] T 9ab933b86de34f0283456231b01db9e5 P 477f6e49554f4d51b8dea60ee5070cb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 778bdad5832d48f287dedf586b21d9ec in term 0.
I20260928 04:45:39.947705 14954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab933b86de34f0283456231b01db9e5" candidate_uuid: "778bdad5832d48f287dedf586b21d9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "477f6e49554f4d51b8dea60ee5070cb7"
I20260928 04:45:39.947768 14954 raft_consensus.cc:3037] T 9ab933b86de34f0283456231b01db9e5 P 477f6e49554f4d51b8dea60ee5070cb7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:45:39.947715 14669 catalog_manager.cc:5495] T 9ab933b86de34f0283456231b01db9e5 P 778bdad5832d48f287dedf586b21d9ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 778bdad5832d48f287dedf586b21d9ec (127.1.226.193). New cstate: current_term: 1 leader_uuid: "778bdad5832d48f287dedf586b21d9ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60a64332b3b844e8bce7c51e5322a287" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "477f6e49554f4d51b8dea60ee5070cb7" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "778bdad5832d48f287dedf586b21d9ec" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 45513 } health_report { overall_health: HEALTHY } } }
I20260928 04:45:39.948625 14954 raft_consensus.cc:2454] T 9ab933b86de34f0283456231b01db9e5 P 477f6e49554f4d51b8dea60ee5070cb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 778bdad5832d48f287dedf586b21d9ec in term 1.
I20260928 04:45:39.965556 15165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:39.962323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38328 (local address 127.1.226.193:45513)
0928 04:45:39.962366 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:39.962367 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:39.962374 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:45:39.962414 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:39.962415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:39.962437 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:39.962496 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:45:39.962665 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:39.963386 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:39.964464 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:39.964776 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:39.964883 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:39.965078 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:45:39.965217 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:45:39.965289 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:39.965451 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 04:45:39.965474 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:45:39.968956 14954 raft_consensus.cc:1261] T 9ab933b86de34f0283456231b01db9e5 P 477f6e49554f4d51b8dea60ee5070cb7 [term 1 FOLLOWER]: Refusing update from remote peer 778bdad5832d48f287dedf586b21d9ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:45:39.969302 15176 consensus_queue.cc:1035] T 9ab933b86de34f0283456231b01db9e5 P 778bdad5832d48f287dedf586b21d9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "477f6e49554f4d51b8dea60ee5070cb7" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 44607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:45:39.971911 15183 mvcc.cc:205] Tried to move back new op lower bound from 7334177750907428864 to 7334177750823043072. Current Snapshot: MvccSnapshot[applied={T|T < 7334177750907428864}]
I20260928 04:45:39.972209 15083 raft_consensus.cc:1261] T 9ab933b86de34f0283456231b01db9e5 P 60a64332b3b844e8bce7c51e5322a287 [term 1 FOLLOWER]: Refusing update from remote peer 778bdad5832d48f287dedf586b21d9ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:45:39.972492 15176 consensus_queue.cc:1035] T 9ab933b86de34f0283456231b01db9e5 P 778bdad5832d48f287dedf586b21d9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "60a64332b3b844e8bce7c51e5322a287" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 34981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:45:39.974241 15187 mvcc.cc:205] Tried to move back new op lower bound from 7334177750907428864 to 7334177750823043072. Current Snapshot: MvccSnapshot[applied={T|T < 7334177750907428864}]
I20260928 04:45:39.976646 15184 mvcc.cc:205] Tried to move back new op lower bound from 7334177750907428864 to 7334177750823043072. Current Snapshot: MvccSnapshot[applied={T|T < 7334177750907428864}]
I20260928 04:45:39.979067 14669 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:45:39.979171 14669 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:45:39.979892 14669 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7608e21f9fd044cb81935eea2da40bc4: Sending DeleteTablet for 3 replicas of tablet 9ab933b86de34f0283456231b01db9e5
I20260928 04:45:39.980366 14935 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ab933b86de34f0283456231b01db9e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:45:39 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:49494
I20260928 04:45:39.980662 15053 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ab933b86de34f0283456231b01db9e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:45:39 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:38782
I20260928 04:45:39.981120 14801 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ab933b86de34f0283456231b01db9e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:45:39 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:38322
I20260928 04:45:39.981362 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 14707
I20260928 04:45:39.988322 15192 tablet_replica.cc:330] T 9ab933b86de34f0283456231b01db9e5 P 477f6e49554f4d51b8dea60ee5070cb7: stopping tablet replica
I20260928 04:45:39.988494 15192 raft_consensus.cc:2229] T 9ab933b86de34f0283456231b01db9e5 P 477f6e49554f4d51b8dea60ee5070cb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:45:39.988648 15192 raft_consensus.cc:2258] T 9ab933b86de34f0283456231b01db9e5 P 477f6e49554f4d51b8dea60ee5070cb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:45:39.988937 15193 tablet_replica.cc:330] T 9ab933b86de34f0283456231b01db9e5 P 60a64332b3b844e8bce7c51e5322a287: stopping tablet replica
I20260928 04:45:39.989064 15193 raft_consensus.cc:2229] T 9ab933b86de34f0283456231b01db9e5 P 60a64332b3b844e8bce7c51e5322a287 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:45:39.989189 15193 raft_consensus.cc:2258] T 9ab933b86de34f0283456231b01db9e5 P 60a64332b3b844e8bce7c51e5322a287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:45:39.989606 15192 ts_tablet_manager.cc:1905] T 9ab933b86de34f0283456231b01db9e5 P 477f6e49554f4d51b8dea60ee5070cb7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 04:45:39.990131 14654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.226.193:45513 (error 108)
I20260928 04:45:39.991043 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:39.990908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:45513 (local address 127.0.0.1:38336)
0928 04:45:39.990968 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:39.991007 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.193:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 04:45:39.991288 14654 catalog_manager.cc:4584] TS 778bdad5832d48f287dedf586b21d9ec (127.1.226.193:45513): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ab933b86de34f0283456231b01db9e5: Network error: Client connection negotiation failed: client connection to 127.1.226.193:45513: connect: Connection refused (error 111)
I20260928 04:45:39.991557 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 14871
I20260928 04:45:39.991570 15193 ts_tablet_manager.cc:1905] T 9ab933b86de34f0283456231b01db9e5 P 60a64332b3b844e8bce7c51e5322a287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:45:39.996030 15193 ts_tablet_manager.cc:1918] T 9ab933b86de34f0283456231b01db9e5 P 60a64332b3b844e8bce7c51e5322a287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:45:39.996119 15193 log.cc:1196] T 9ab933b86de34f0283456231b01db9e5 P 60a64332b3b844e8bce7c51e5322a287: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570585693787-1931-0/minicluster-data/ts-2/wal/wals/9ab933b86de34f0283456231b01db9e5
I20260928 04:45:39.996423 15193 ts_tablet_manager.cc:1939] T 9ab933b86de34f0283456231b01db9e5 P 60a64332b3b844e8bce7c51e5322a287: Deleting consensus metadata
I20260928 04:45:40.000808 14655 catalog_manager.cc:4841] TS 60a64332b3b844e8bce7c51e5322a287 (127.1.226.195:34981): tablet 9ab933b86de34f0283456231b01db9e5 (table test-table [id=29283e6712b24c7f89e005a4d4741f87]) successfully deleted
I20260928 04:45:40.001641 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:40.001520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:44607 (local address 127.0.0.1:49498)
0928 04:45:40.001566 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:40.001597 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.194:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 04:45:40.001677 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 15004
I20260928 04:45:40.008047 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 14636
I20260928 04:45:40.019253 1931 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 04:45:40.364861 1931 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 04:45:40.364975 1931 mini_ranger.cc:66] Stopping Ranger...
I20260928 04:45:40.709550 1931 mini_ranger.cc:68] Stopped Ranger
2026-09-28 04:45:40.709 UTC [13703] LOG: received smart shutdown request
2026-09-28 04:45:40.714 UTC [13703] LOG: background worker "logical replication launcher" (PID 13712) exited with exit code 1
2026-09-28 04:45:40.714 UTC [13707] LOG: shutting down
2026-09-28 04:45:40.895 UTC [13703] LOG: database system is shut down
2026-09-28T04:45:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48761 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:45:41 dist-test-slave-j5c3 krb5kdc[15205](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:45:41 dist-test-slave-j5c3 krb5kdc[15205](info): set up 2 sockets
Sep 28 04:45:41 dist-test-slave-j5c3 krb5kdc[15205](info): commencing operation
krb5kdc: starting...
W20260928 04:45:42.096100 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:45:42 dist-test-slave-j5c3 krb5kdc[15205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:45:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:45:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:45:42.215979 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:38645
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:38645,127.1.226.253:43943,127.1.226.252:45569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:45:42.298743 15218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:45:42.298978 15218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:45:42.299002 15218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:45:42.300138 15218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:45:42.300207 15218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:45:42.300221 15218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:45:42.300235 15218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:45:42.300246 15218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:45:42.302048 15218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:38645,127.1.226.253:43943,127.1.226.252:45569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:38645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:45:42.302335 15218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:45:42.302572 15218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:45:42.307642 15224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:42.307757 15223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:42.307775 15226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:42.307837 15218 server_base.cc:1034] running on GCE node
I20260928 04:45:42.308357 15218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:45:42.308652 15218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:45:42.309813 15218 hybrid_clock.cc:648] HybridClock initialized: now 1790570742309788 us; error 40 us; skew 500 ppm
Sep 28 04:45:42 dist-test-slave-j5c3 krb5kdc[15205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570742, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:45:42.312398 15218 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:45:42.312822 15218 webserver.cc:460] Webserver started at http://127.1.226.254:45615/ using document root <none> and password file <none>
I20260928 04:45:42.313005 15218 fs_manager.cc:359] Metadata directory not provided
I20260928 04:45:42.313045 15218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:45:42.313135 15218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:45:42.314136 15218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "fec47c0dfd974f658d99c533daa92f85"
format_stamp: "Formatted at 2026-09-28 04:45:42 on dist-test-slave-j5c3"
server_key: "a5a0a966100806279d5e404c7b4b0e05"
server_key_iv: "6d95c9485c4f2d947593c834e4a09ca6"
server_key_version: "encryptionkey@0"
I20260928 04:45:42.314445 15218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "fec47c0dfd974f658d99c533daa92f85"
format_stamp: "Formatted at 2026-09-28 04:45:42 on dist-test-slave-j5c3"
server_key: "a5a0a966100806279d5e404c7b4b0e05"
server_key_iv: "6d95c9485c4f2d947593c834e4a09ca6"
server_key_version: "encryptionkey@0"
I20260928 04:45:42.315675 15218 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 04:45:42.316367 15233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:42.316536 15218 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:42.316639 15218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "fec47c0dfd974f658d99c533daa92f85"
format_stamp: "Formatted at 2026-09-28 04:45:42 on dist-test-slave-j5c3"
server_key: "a5a0a966100806279d5e404c7b4b0e05"
server_key_iv: "6d95c9485c4f2d947593c834e4a09ca6"
server_key_version: "encryptionkey@0"
I20260928 04:45:42.316725 15218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:45:42.333053 15218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:45:42.334952 15218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:45:42.335124 15218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:45:42.339669 15218 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:38645
I20260928 04:45:42.339704 15285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:38645 every 8 connection(s)
I20260928 04:45:42.340087 15218 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:45:42.340890 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 15218
I20260928 04:45:42.341023 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:45:42.341100 15286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:45:42.341240 1931 external_mini_cluster.cc:1428] Setting key 8f8a834c3a222c0db7746a665161242f
I20260928 04:45:42.342705 15286 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38645 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.226.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.253@KRBTEST.COM" created.
Sep 28 04:45:42 dist-test-slave-j5c3 krb5kdc[15205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570742, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:45:42.355207 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.343415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38645 (local address 127.0.0.1:53986)
0928 04:45:42.343602 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:42.343654 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:42.344226 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:42.344556 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:42.344559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:42.344851 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:45:42.345015 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:42.345020 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.346089 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.346090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:42.346781 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:42.346784 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.347338 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.347339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:42.347405 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:42.347971 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:45:42.347979 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:45:42.351760 (+ 3781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:45:42.353745 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:42.353751 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:42.353753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:42.353936 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:42.354177 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:42.354178 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:42.354179 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:42.354267 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:42.354616 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:45:42.354620 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:45:42.354681 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 04:45:42.354787 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 04:45:42.355019 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":94,"thread_start_us":97,"threads_started":1}
I20260928 04:45:42.355207 15290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.343431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53986 (local address 127.1.226.254:38645)
0928 04:45:42.343648 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:42.343659 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:42.343766 (+ 107us) server_negotiation.cc:407] Connection header received
0928 04:45:42.344398 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:42.344400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:42.344431 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:42.344550 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:42.345125 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.345967 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.346897 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.347232 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.347366 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:42.351875 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:42.351882 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:42.351883 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:42.351890 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:42.353640 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.354029 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.354032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.354032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.354111 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.354331 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.354332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.354333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.354546 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:42.354637 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:42.354813 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 04:45:42.354955 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":123,"threads_started":1}
I20260928 04:45:42.355901 15286 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.253" port: 43943 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:45:42.356261 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.356131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:43943 (local address 127.0.0.1:47096)
0928 04:45:42.356193 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:42.356237 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.253:43943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 04:45:42.356380 15236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.226.253:43943: connect: Connection refused (error 111)
I20260928 04:45:42.356801 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.356711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:43943 (local address 127.0.0.1:47110)
0928 04:45:42.356757 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:42.356780 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.253:43943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 04:45:42.356902 15286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.226.253:43943: Network error: Client connection negotiation failed: client connection to 127.1.226.253:43943: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.253.keytab.
Entry for principal kudu/127.1.226.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.253.keytab.
I20260928 04:45:42.367167 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.253:43943
--webserver_interface=127.1.226.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.253.keytab
--principal=kudu/127.1.226.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:38645,127.1.226.253:43943,127.1.226.252:45569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:45:42.406132 15286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.226.253" port: 43943 } attempt: 1
I20260928 04:45:42.406726 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.406584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:43943 (local address 127.0.0.1:47116)
0928 04:45:42.406653 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:42.406697 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.253:43943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 04:45:42.407115 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.407036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:43943 (local address 127.0.0.1:47118)
0928 04:45:42.407074 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:42.407100 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.253:43943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 04:45:42.407217 15286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.226.253:43943: Network error: Client connection negotiation failed: client connection to 127.1.226.253:43943: connect: Connection refused (error 111)
W20260928 04:45:42.451102 15293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:45:42.451315 15293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:45:42.451339 15293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:45:42.452616 15293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:45:42.452670 15293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:45:42.452687 15293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:45:42.452702 15293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:45:42.452714 15293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:45:42.454173 15293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:38645,127.1.226.253:43943,127.1.226.252:45569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.253.keytab
--principal=kudu/127.1.226.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.253:43943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.226.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:45:42.454373 15293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:45:42.454569 15293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:45:42.459972 15301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:42.459990 15299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:42.459972 15298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:42.460125 15293 server_base.cc:1034] running on GCE node
I20260928 04:45:42.460522 15293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:45:42.460819 15293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:45:42.461988 15293 hybrid_clock.cc:648] HybridClock initialized: now 1790570742461968 us; error 40 us; skew 500 ppm
Sep 28 04:45:42 dist-test-slave-j5c3 krb5kdc[15205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570742, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:45:42.464722 15293 init.cc:377] Logged in from keytab as kudu/127.1.226.253@KRBTEST.COM (short username kudu)
I20260928 04:45:42.465134 15293 webserver.cc:460] Webserver started at http://127.1.226.253:33653/ using document root <none> and password file <none>
I20260928 04:45:42.465395 15293 fs_manager.cc:359] Metadata directory not provided
I20260928 04:45:42.465446 15293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:45:42.465584 15293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:45:42.466667 15293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-1/data/instance:
uuid: "65a485a2db7040fd9f7a0c07ca2ee9b9"
format_stamp: "Formatted at 2026-09-28 04:45:42 on dist-test-slave-j5c3"
server_key: "e32735945c671a43108aa5ee8126969e"
server_key_iv: "67dd8d46a53bf68487728a672c2c8274"
server_key_version: "encryptionkey@0"
I20260928 04:45:42.467041 15293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-1/wal/instance:
uuid: "65a485a2db7040fd9f7a0c07ca2ee9b9"
format_stamp: "Formatted at 2026-09-28 04:45:42 on dist-test-slave-j5c3"
server_key: "e32735945c671a43108aa5ee8126969e"
server_key_iv: "67dd8d46a53bf68487728a672c2c8274"
server_key_version: "encryptionkey@0"
I20260928 04:45:42.468518 15293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260928 04:45:42.469461 15308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:42.469748 15293 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:45:42.469872 15293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-1/wal
uuid: "65a485a2db7040fd9f7a0c07ca2ee9b9"
format_stamp: "Formatted at 2026-09-28 04:45:42 on dist-test-slave-j5c3"
server_key: "e32735945c671a43108aa5ee8126969e"
server_key_iv: "67dd8d46a53bf68487728a672c2c8274"
server_key_version: "encryptionkey@0"
I20260928 04:45:42.469952 15293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:45:42.475404 15286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.226.253" port: 43943 } attempt: 2
I20260928 04:45:42.475863 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.475732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:43943 (local address 127.0.0.1:47130)
0928 04:45:42.475793 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:42.475835 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.253:43943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 04:45:42.476269 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.476134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:43943 (local address 127.0.0.1:47142)
0928 04:45:42.476204 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:42.476236 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.253:43943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 04:45:42.476423 15286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.226.253:43943: Network error: Client connection negotiation failed: client connection to 127.1.226.253:43943: connect: Connection refused (error 111)
I20260928 04:45:42.500383 15293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:45:42.502503 15293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:45:42.502663 15293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:45:42.506810 15293 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.253:43943
I20260928 04:45:42.506843 15360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.253:43943 every 8 connection(s)
I20260928 04:45:42.507244 15293 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-1/data/info.pb
I20260928 04:45:42.508143 15361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:45:42.509752 15361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38645 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:45:42.512799 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 15293
I20260928 04:45:42.512919 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-1/wal/instance
I20260928 04:45:42.513127 1931 external_mini_cluster.cc:1428] Setting key c90d1fbe764d30693aa08fc4ab0cbcb4
Sep 28 04:45:42 dist-test-slave-j5c3 krb5kdc[15205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570742, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.253@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:45:42.520851 15290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.510499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53996 (local address 127.1.226.254:38645)
0928 04:45:42.510562 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:42.510564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:42.510770 (+ 206us) server_negotiation.cc:407] Connection header received
0928 04:45:42.511379 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:42.511381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:42.511394 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:42.511478 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:42.512328 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.513095 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.514597 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.514948 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.514995 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:42.517550 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:42.517552 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:42.517552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:42.517561 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:42.519223 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.519689 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.519690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.519691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.519766 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.520061 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.520062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.520062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.520235 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:42.520314 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:42.520609 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 04:45:42.520732 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 04:45:42.520967 15363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.510489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38645 (local address 127.0.0.1:53996)
0928 04:45:42.510640 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:42.510681 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:42.511264 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:42.511550 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:42.511554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:42.511886 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:45:42.512193 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:42.512200 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.513195 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.513196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:42.514480 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:42.514484 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.515092 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.515093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:42.515246 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:42.515773 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:45:42.515783 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:45:42.517451 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:45:42.519323 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:42.519331 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:42.519333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:42.519609 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:42.519871 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:42.519872 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:42.519872 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:42.519982 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:42.520314 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:45:42.520320 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:45:42.520475 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 04:45:42.520565 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 04:45:42.520728 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.1.226.252@KRBTEST.COM; defaulting to no policy
I20260928 04:45:42.521509 15361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.253" port: 43943 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.252@KRBTEST.COM" created.
Sep 28 04:45:42 dist-test-slave-j5c3 krb5kdc[15205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570742, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.253@KRBTEST.COM for kudu/127.1.226.253@KRBTEST.COM
I20260928 04:45:42.528963 15363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.521804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:43943 (local address 127.0.0.1:47152)
0928 04:45:42.521854 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:42.521865 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:42.521907 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:42.522154 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:42.522155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:42.522298 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:45:42.522340 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:42.522342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.523205 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.523206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:42.523697 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:42.523698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.524228 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.524229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:42.524301 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:42.524670 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:45:42.524675 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:45:42.526025 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:45:42.527874 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:42.527875 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:42.527876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:42.528045 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:42.528280 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:42.528281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:42.528281 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:42.528334 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:42.528622 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:45:42.528623 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:45:42.528661 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 04:45:42.528723 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 04:45:42.528831 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 04:45:42.528995 15365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.521871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47152 (local address 127.1.226.253:43943)
0928 04:45:42.522017 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:42.522022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:42.522046 (+ 24us) server_negotiation.cc:407] Connection header received
0928 04:45:42.522063 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:42.522065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:42.522075 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:42.522154 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:42.522421 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.523103 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.523794 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.524131 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.524230 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:42.526112 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:42.526121 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:42.526121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:42.526129 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:42.527785 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.528126 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.528129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.528130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.528206 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.528409 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.528409 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.528410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.528552 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:42.528644 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:42.528762 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 04:45:42.528885 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260928 04:45:42.529897 15361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45569 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:45:42.530273 15363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.530150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:45569 (local address 127.0.0.1:43700)
0928 04:45:42.530199 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:42.530244 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:45569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 04:45:42.530402 15311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.226.252:45569: connect: Connection refused (error 111)
I20260928 04:45:42.530817 15363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.530713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:45569 (local address 127.0.0.1:43716)
0928 04:45:42.530764 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:42.530795 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:45569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 04:45:42.530920 15361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.226.252:45569: Network error: Client connection negotiation failed: client connection to 127.1.226.252:45569: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.252.keytab.
Entry for principal kudu/127.1.226.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.252.keytab.
I20260928 04:45:42.539481 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.252:45569
--webserver_interface=127.1.226.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.252.keytab
--principal=kudu/127.1.226.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:38645,127.1.226.253:43943,127.1.226.252:45569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:45:42.567675 15286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.226.253" port: 43943 } attempt: 3
Sep 28 04:45:42 dist-test-slave-j5c3 krb5kdc[15205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570742, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for kudu/127.1.226.253@KRBTEST.COM
I20260928 04:45:42.575028 15365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.568229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47158 (local address 127.1.226.253:43943)
0928 04:45:42.568306 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:42.568308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:42.568319 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:45:42.568408 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:42.568409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:42.568426 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:42.568487 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:42.568838 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.569331 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.570045 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.570258 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.570339 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:42.572278 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:42.572279 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:42.572280 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:42.572293 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:42.573859 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.574200 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.574201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.574202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.574263 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.574480 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.574482 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.574482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.574600 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:42.574675 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:42.574849 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 04:45:42.574923 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 04:45:42.575028 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.568120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:43943 (local address 127.0.0.1:47158)
0928 04:45:42.568197 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:42.568210 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:42.568286 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:42.568509 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:42.568510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:42.568676 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:45:42.568741 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:42.568743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.569430 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.569431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:42.569955 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:42.569956 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.570338 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.570339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:42.570422 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:42.570928 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:45:42.570933 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:45:42.572141 (+ 1208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:45:42.573946 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:42.573947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:42.573948 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:42.574125 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:42.574348 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:42.574349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:42.574349 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:42.574412 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:42.574680 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:45:42.574681 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:45:42.574731 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 04:45:42.574816 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 04:45:42.574923 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 04:45:42.575472 15286 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45569 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:45:42.575867 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.575727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:45569 (local address 127.0.0.1:43720)
0928 04:45:42.575770 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:42.575835 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:45569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 04:45:42.576196 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.576090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:45569 (local address 127.0.0.1:43726)
0928 04:45:42.576135 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:42.576175 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:45569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 04:45:42.576295 15286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.226.252:45569: Network error: Client connection negotiation failed: client connection to 127.1.226.252:45569: connect: Connection refused (error 111)
I20260928 04:45:42.580210 15361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45569 } attempt: 1
I20260928 04:45:42.580639 15363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.580517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:45569 (local address 127.0.0.1:43734)
0928 04:45:42.580573 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:42.580603 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:45569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 04:45:42.581002 15363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.580906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:45569 (local address 127.0.0.1:43746)
0928 04:45:42.580953 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:42.580978 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:45569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 04:45:42.581095 15361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.226.252:45569: Network error: Client connection negotiation failed: client connection to 127.1.226.252:45569: connect: Connection refused (error 111)
I20260928 04:45:42.607447 15286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45569 } attempt: 1
I20260928 04:45:42.607980 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.607818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:45569 (local address 127.0.0.1:43750)
0928 04:45:42.607907 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:42.607949 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:45569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 04:45:42.608371 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.608270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:45569 (local address 127.0.0.1:43762)
0928 04:45:42.608319 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:42.608351 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:45569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 04:45:42.608461 15286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.226.252:45569: Network error: Client connection negotiation failed: client connection to 127.1.226.252:45569: connect: Connection refused (error 111)
W20260928 04:45:42.626533 15368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:45:42.626742 15368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:45:42.626765 15368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:45:42.627892 15368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:45:42.627933 15368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:45:42.627947 15368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:45:42.627959 15368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:45:42.628011 15368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:45:42.629580 15368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:38645,127.1.226.253:43943,127.1.226.252:45569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.252.keytab
--principal=kudu/127.1.226.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.252:45569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.226.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:45:42.629880 15368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:45:42.630128 15368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:45:42.635015 15374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:42.635056 15373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:42.635090 15376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:42.635406 15368 server_base.cc:1034] running on GCE node
I20260928 04:45:42.635658 15368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:45:42.636044 15368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:45:42.637195 15368 hybrid_clock.cc:648] HybridClock initialized: now 1790570742637154 us; error 66 us; skew 500 ppm
Sep 28 04:45:42 dist-test-slave-j5c3 krb5kdc[15205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570742, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:45:42.639863 15368 init.cc:377] Logged in from keytab as kudu/127.1.226.252@KRBTEST.COM (short username kudu)
I20260928 04:45:42.640420 15368 webserver.cc:460] Webserver started at http://127.1.226.252:39709/ using document root <none> and password file <none>
I20260928 04:45:42.640666 15368 fs_manager.cc:359] Metadata directory not provided
I20260928 04:45:42.640733 15368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:45:42.640887 15368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:45:42.641953 15368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-2/data/instance:
uuid: "90afcd51275f414da130bea3e56cd5f4"
format_stamp: "Formatted at 2026-09-28 04:45:42 on dist-test-slave-j5c3"
server_key: "098c051d6405ae96bf0e9b283079480a"
server_key_iv: "e16fc9e159b1362c10ffd7b870dba28b"
server_key_version: "encryptionkey@0"
I20260928 04:45:42.642314 15368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-2/wal/instance:
uuid: "90afcd51275f414da130bea3e56cd5f4"
format_stamp: "Formatted at 2026-09-28 04:45:42 on dist-test-slave-j5c3"
server_key: "098c051d6405ae96bf0e9b283079480a"
server_key_iv: "e16fc9e159b1362c10ffd7b870dba28b"
server_key_version: "encryptionkey@0"
I20260928 04:45:42.643745 15368 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:45:42.644618 15383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:42.644841 15368 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:45:42.644946 15368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-2/wal
uuid: "90afcd51275f414da130bea3e56cd5f4"
format_stamp: "Formatted at 2026-09-28 04:45:42 on dist-test-slave-j5c3"
server_key: "098c051d6405ae96bf0e9b283079480a"
server_key_iv: "e16fc9e159b1362c10ffd7b870dba28b"
server_key_version: "encryptionkey@0"
I20260928 04:45:42.645026 15368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:45:42.649250 15361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45569 } attempt: 2
I20260928 04:45:42.649780 15363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.649558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:45569 (local address 127.0.0.1:43778)
0928 04:45:42.649643 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:42.649680 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:45569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 04:45:42.650185 15363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.650078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:45569 (local address 127.0.0.1:43786)
0928 04:45:42.650128 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:42.650164 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:45569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 04:45:42.650285 15361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.226.252:45569: Network error: Client connection negotiation failed: client connection to 127.1.226.252:45569: connect: Connection refused (error 111)
I20260928 04:45:42.670514 15368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:45:42.672868 15368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:45:42.673030 15368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:45:42.677601 15368 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.252:45569
I20260928 04:45:42.677637 15435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.252:45569 every 8 connection(s)
I20260928 04:45:42.678040 15368 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-2/data/info.pb
I20260928 04:45:42.678994 15436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:45:42.680526 15436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38645 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:45:42.683609 15286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45569 } attempt: 2
I20260928 04:45:42.684890 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 15368
I20260928 04:45:42.685010 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570585693787-1931-0/minicluster-data/master-2/wal/instance
I20260928 04:45:42.685209 1931 external_mini_cluster.cc:1428] Setting key 23a62f374e2f84bc9524b1021a536220
Sep 28 04:45:42 dist-test-slave-j5c3 krb5kdc[15205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570742, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.252@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
Sep 28 04:45:42 dist-test-slave-j5c3 krb5kdc[15205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570742, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for kudu/127.1.226.252@KRBTEST.COM
Sep 28 04:45:42 dist-test-slave-j5c3 krb5kdc[15205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.252@KRBTEST.COM
I20260928 04:45:42.691344 15290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.681254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54012 (local address 127.1.226.254:38645)
0928 04:45:42.681348 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:42.681350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:42.681537 (+ 187us) server_negotiation.cc:407] Connection header received
0928 04:45:42.682098 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:42.682099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:42.682111 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:42.682169 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:42.682824 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.683305 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.684527 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.684804 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.684901 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:42.687711 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:42.687713 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:42.687713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:42.687721 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:42.689527 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.690069 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.690071 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.690071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.690133 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.690461 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.690463 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.690463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.690717 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:42.690783 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:42.691150 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 04:45:42.691228 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 04:45:42.691582 15438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.681219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38645 (local address 127.0.0.1:54012)
0928 04:45:42.681410 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:42.681448 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:42.681981 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:42.682250 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:42.682255 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:42.682513 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:45:42.682714 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:42.682718 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.683407 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.683408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:42.684430 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:42.684434 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.684910 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.684912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:42.685078 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:42.685774 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:45:42.685784 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:45:42.687600 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:45:42.689631 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:42.689640 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:42.689642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:42.689970 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:42.690208 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:42.690208 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:42.690209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:42.690367 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:42.690780 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:45:42.690787 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:45:42.690969 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 04:45:42.691120 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 04:45:42.691319 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":84,"threads_started":1}
I20260928 04:45:42.692230 15436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.253" port: 43943 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:45:42.693486 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.683993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:45569 (local address 127.0.0.1:43802)
0928 04:45:42.684057 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:42.684070 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:42.684116 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:42.684429 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:42.684430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:42.684598 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:45:42.684634 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:42.684635 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.685769 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.685770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:42.686539 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:42.686542 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.687247 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.687248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:42.687313 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:42.687789 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:45:42.687794 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:45:42.689417 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:45:42.691845 (+ 2428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:42.691846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:42.691847 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:42.692093 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:42.692625 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:42.692626 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:42.692627 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:42.692718 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:42.693158 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:45:42.693159 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:45:42.693210 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 04:45:42.693278 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 04:45:42.693380 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 04:45:42.694087 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.684051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43802 (local address 127.1.226.252:45569)
0928 04:45:42.684242 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:42.684247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:42.684270 (+ 23us) server_negotiation.cc:407] Connection header received
0928 04:45:42.684296 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:42.684299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:42.684332 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:42.684407 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:42.684711 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.685671 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.686667 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.687146 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.687486 (+ 340us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:42.689516 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:42.689525 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:42.689526 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:42.689532 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:42.691720 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.692228 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.692231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.692232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.692312 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.692822 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.692823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.692824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.693048 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:42.693885 (+ 837us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:42.693947 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 04:45:42.694010 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":165,"threads_started":1}
I20260928 04:45:42.695320 15440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.685545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43806 (local address 127.1.226.252:45569)
0928 04:45:42.685826 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:42.685827 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:42.685837 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:45:42.685864 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:42.685864 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:42.685876 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:42.685969 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:42.686615 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.687073 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.688061 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.688350 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.688457 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:42.691124 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:42.691126 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:42.691126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:42.691141 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:42.693794 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.694201 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.694202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.694202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.694267 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.694467 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.694468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.694468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.694596 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:42.694725 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:42.695135 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 04:45:42.695203 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":98,"threads_started":1}
I20260928 04:45:42.696015 15286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fec47c0dfd974f658d99c533daa92f85: Bootstrap starting.
Sep 28 04:45:42 dist-test-slave-j5c3 krb5kdc[15205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570742, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.252@KRBTEST.COM for kudu/127.1.226.253@KRBTEST.COM
I20260928 04:45:42.697091 15286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fec47c0dfd974f658d99c533daa92f85: Neither blocks nor log segments found. Creating new log.
I20260928 04:45:42.697446 15286 log.cc:824] T 00000000000000000000000000000000 P fec47c0dfd974f658d99c533daa92f85: Log is configured to *not* fsync() on all Append() calls
I20260928 04:45:42.698537 15286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fec47c0dfd974f658d99c533daa92f85: No bootstrap required, opened a new log
I20260928 04:45:42.699920 15365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.692627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47168 (local address 127.1.226.253:43943)
0928 04:45:42.692672 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:42.692675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:42.692683 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:45:42.692708 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:42.692709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:42.692718 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:42.692767 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:42.693033 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.693501 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.694430 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.694757 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.694865 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:42.696940 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:42.696942 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:42.696942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:42.696949 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:42.698782 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.699143 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.699144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.699144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.699197 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.699432 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.699434 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.699434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.699568 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:42.699633 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:42.699798 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 04:45:42.699861 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 04:45:42.699963 15438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.692535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:43943 (local address 127.0.0.1:47168)
0928 04:45:42.692596 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:42.692609 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:42.692653 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:42.692769 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:42.692769 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:42.692904 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:45:42.692952 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:42.692954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.693583 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.693585 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:42.694316 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:42.694318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.694862 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.694863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:42.694939 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:42.695384 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:45:42.695390 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:45:42.696849 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:45:42.698861 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:42.698862 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:42.698862 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:42.699066 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:42.699266 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:42.699267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:42.699267 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:42.699333 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:42.699645 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:45:42.699646 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:45:42.699690 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 04:45:42.699781 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 04:45:42.699871 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 04:45:42.700312 15436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45569 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:45:42.701359 15286 raft_consensus.cc:348] T 00000000000000000000000000000000 P fec47c0dfd974f658d99c533daa92f85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec47c0dfd974f658d99c533daa92f85" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38645 } } peers { permanent_uuid: "65a485a2db7040fd9f7a0c07ca2ee9b9" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 43943 } } peers { permanent_uuid: "90afcd51275f414da130bea3e56cd5f4" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45569 } }
I20260928 04:45:42.701568 15286 raft_consensus.cc:374] T 00000000000000000000000000000000 P fec47c0dfd974f658d99c533daa92f85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:45:42.701668 15286 raft_consensus.cc:729] T 00000000000000000000000000000000 P fec47c0dfd974f658d99c533daa92f85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec47c0dfd974f658d99c533daa92f85, State: Initialized, Role: FOLLOWER
I20260928 04:45:42.701865 15286 consensus_queue.cc:260] T 00000000000000000000000000000000 P fec47c0dfd974f658d99c533daa92f85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec47c0dfd974f658d99c533daa92f85" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38645 } } peers { permanent_uuid: "65a485a2db7040fd9f7a0c07ca2ee9b9" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 43943 } } peers { permanent_uuid: "90afcd51275f414da130bea3e56cd5f4" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45569 } }
I20260928 04:45:42.702193 15286 sys_catalog.cc:564] T 00000000000000000000000000000000 P fec47c0dfd974f658d99c533daa92f85 [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 04:45:42 dist-test-slave-j5c3 krb5kdc[15205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570742, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.252@KRBTEST.COM for kudu/127.1.226.252@KRBTEST.COM
Sep 28 04:45:42 dist-test-slave-j5c3 krb5kdc[15205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
W20260928 04:45:42.706039 15454 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fec47c0dfd974f658d99c533daa92f85: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:45:42.706135 15454 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:45:42.705441 15443 sys_catalog.cc:455] T 00000000000000000000000000000000 P fec47c0dfd974f658d99c533daa92f85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec47c0dfd974f658d99c533daa92f85" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38645 } } peers { permanent_uuid: "65a485a2db7040fd9f7a0c07ca2ee9b9" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 43943 } } peers { permanent_uuid: "90afcd51275f414da130bea3e56cd5f4" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45569 } } }
I20260928 04:45:42.706357 15443 sys_catalog.cc:458] T 00000000000000000000000000000000 P fec47c0dfd974f658d99c533daa92f85 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:45:42.708570 15438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.700678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:45569 (local address 127.0.0.1:43812)
0928 04:45:42.700723 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:42.700781 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:42.700836 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:42.700987 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:42.700988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:42.701130 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:45:42.701156 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:42.701157 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.701864 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.701865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:42.702595 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:42.702598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.703107 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.703108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:42.703164 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:42.703616 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:45:42.703621 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:45:42.705014 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:45:42.707431 (+ 2417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:42.707431 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:42.707432 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:42.707601 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:42.707810 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:42.707811 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:42.707811 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:42.707873 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:42.708200 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:45:42.708201 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:45:42.708244 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 04:45:42.708336 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 04:45:42.708450 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 04:45:42.708626 15440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.700710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43812 (local address 127.1.226.252:45569)
0928 04:45:42.700748 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:42.700749 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:42.700839 (+ 90us) server_negotiation.cc:407] Connection header received
0928 04:45:42.700906 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:42.700907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:42.700917 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:42.700983 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:42.701268 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.701762 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.702699 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.703012 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.703108 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:42.705117 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:42.705118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:42.705119 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:42.705126 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:42.707333 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.707676 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.707677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.707678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.707737 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.707942 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.707943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.707943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.708087 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:42.708200 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:42.708372 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 04:45:42.708505 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":21}
I20260928 04:45:42.707072 15218 master_runner.cc:186] Error getting master registration for 127.1.226.253:43943: OK, instance_id { permanent_uuid: "65a485a2db7040fd9f7a0c07ca2ee9b9" instance_seqno: 1790570742505592 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 04:45:42.709964 15218 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 04:45:42.712136 15290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.698431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.1.226.254:38645)
0928 04:45:42.700122 (+ 1691us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:42.700124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:42.700130 (+ 6us) server_negotiation.cc:407] Connection header received
0928 04:45:42.700146 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:42.700147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:42.700197 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:42.700264 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:42.700867 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.701540 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.702515 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.702829 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.702877 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:42.705678 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:42.705679 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:42.705680 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:42.705686 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:42.710778 (+ 5092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.711328 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.711329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.711330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.711395 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.711601 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.711602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.711602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.711731 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:42.711921 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:42.711961 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 04:45:42.712038 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1673}
I20260928 04:45:42.712452 15436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4: Bootstrap starting.
I20260928 04:45:42.713195 15436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4: Neither blocks nor log segments found. Creating new log.
I20260928 04:45:42.713557 15436 log.cc:824] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4: Log is configured to *not* fsync() on all Append() calls
I20260928 04:45:42.714502 15436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4: No bootstrap required, opened a new log
I20260928 04:45:42.716519 15436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec47c0dfd974f658d99c533daa92f85" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38645 } } peers { permanent_uuid: "65a485a2db7040fd9f7a0c07ca2ee9b9" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 43943 } } peers { permanent_uuid: "90afcd51275f414da130bea3e56cd5f4" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45569 } }
I20260928 04:45:42.716677 15436 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:45:42.716746 15436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90afcd51275f414da130bea3e56cd5f4, State: Initialized, Role: FOLLOWER
I20260928 04:45:42.716895 15436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec47c0dfd974f658d99c533daa92f85" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38645 } } peers { permanent_uuid: "65a485a2db7040fd9f7a0c07ca2ee9b9" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 43943 } } peers { permanent_uuid: "90afcd51275f414da130bea3e56cd5f4" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45569 } }
I20260928 04:45:42.717257 15436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:45:42.717200 15457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec47c0dfd974f658d99c533daa92f85" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38645 } } peers { permanent_uuid: "65a485a2db7040fd9f7a0c07ca2ee9b9" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 43943 } } peers { permanent_uuid: "90afcd51275f414da130bea3e56cd5f4" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45569 } } }
I20260928 04:45:42.719384 15457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4 [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 04:45:42 dist-test-slave-j5c3 krb5kdc[15205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.253@KRBTEST.COM
W20260928 04:45:42.720432 15468 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:45:42.720530 15468 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:45:42.721812 15368 master_runner.cc:186] Error getting master registration for 127.1.226.253:43943: OK, instance_id { permanent_uuid: "65a485a2db7040fd9f7a0c07ca2ee9b9" instance_seqno: 1790570742505592 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 04:45:42.725118 15368 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 04:45:42.725241 15365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.714458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47170 (local address 127.1.226.253:43943)
0928 04:45:42.714499 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:42.714501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:42.714512 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:45:42.714555 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:42.714556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:42.714566 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:42.714613 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:42.715325 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.716055 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.717178 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.717487 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.717564 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:42.720863 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:42.720865 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:42.720865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:42.720874 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:42.722864 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.723310 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.723311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.723311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.723384 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.723607 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.723607 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.723608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.723757 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:42.724509 (+ 752us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:42.725030 (+ 521us) server_negotiation.cc:299] Negotiation successful
0928 04:45:42.725119 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:45:42.726681 1931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:45:42.741436 15290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.728480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54040 (local address 127.1.226.254:38645)
0928 04:45:42.728526 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:42.728528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:42.732430 (+ 3902us) server_negotiation.cc:407] Connection header received
0928 04:45:42.732532 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:42.732534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:42.732549 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:42.732614 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:42.733508 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.734281 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.735365 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.735656 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.735705 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:42.737481 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:42.737482 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:42.737483 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:42.737492 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:42.739680 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.740429 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.740431 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.740431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.740496 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.740733 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.740734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.740734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.740885 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:42.740984 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:42.741238 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 04:45:42.741330 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:45:42.741411 15361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45569 } attempt: 3
I20260928 04:45:42.747434 15365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.728836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47186 (local address 127.1.226.253:43943)
0928 04:45:42.728879 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:42.728881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:42.732626 (+ 3745us) server_negotiation.cc:407] Connection header received
0928 04:45:42.732824 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:42.732826 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:42.732838 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:42.732873 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:42.734713 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.735384 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.738177 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.738513 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.738571 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:42.741743 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:42.741745 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:42.741746 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:42.741754 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:42.743761 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.745691 (+ 1930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.745692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.745692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.745755 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.746144 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.746145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.746145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.746280 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:42.746381 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:42.747232 (+ 851us) server_negotiation.cc:299] Negotiation successful
0928 04:45:42.747317 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 04:45:42 dist-test-slave-j5c3 krb5kdc[15205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570742, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.253@KRBTEST.COM for kudu/127.1.226.252@KRBTEST.COM
I20260928 04:45:42.748018 15440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.729018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43814 (local address 127.1.226.252:45569)
0928 04:45:42.729060 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:42.729062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:42.733127 (+ 4065us) server_negotiation.cc:407] Connection header received
0928 04:45:42.733837 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:42.733839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:42.733853 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:42.733888 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:42.736364 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.737142 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.739004 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.739311 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.739364 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:42.742747 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:42.742749 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:42.742749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:42.742757 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:42.745089 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.745984 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.745986 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.745986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.746047 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.746766 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.746767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.746767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.746915 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:42.747001 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:42.747822 (+ 821us) server_negotiation.cc:299] Negotiation successful
0928 04:45:42.747904 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:45:42.752264 15363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.741833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:45569 (local address 127.0.0.1:43820)
0928 04:45:42.742322 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:42.742335 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:42.742385 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:42.743254 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:42.743255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:42.743410 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:45:42.743460 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:42.743462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.744756 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.744757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:42.745568 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:42.745571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.746143 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.746144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:42.746202 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:42.746627 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:45:42.746632 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:45:42.749184 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:45:42.750996 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:42.750997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:42.750997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:42.751184 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:42.751432 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:42.751433 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:42.751433 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:42.751498 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:42.751862 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:45:42.751862 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:45:42.751911 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 04:45:42.751990 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 04:45:42.752093 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466}
I20260928 04:45:42.752928 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:42.742493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43820 (local address 127.1.226.252:45569)
0928 04:45:42.742547 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:42.742549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:42.742556 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:45:42.742812 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:42.742813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:42.742825 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:42.742885 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:42.743568 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.744652 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.745665 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:42.746012 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:42.746079 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:42.749293 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:42.749294 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:42.749294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:42.749301 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:42.750880 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.751287 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.751288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.751289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.751348 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:42.751616 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:42.751617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:42.751618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:42.751756 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:42.751821 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:42.752742 (+ 921us) server_negotiation.cc:299] Negotiation successful
0928 04:45:42.752832 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 04:45:42.754262 15361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65a485a2db7040fd9f7a0c07ca2ee9b9: Bootstrap starting.
I20260928 04:45:42.754957 15361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65a485a2db7040fd9f7a0c07ca2ee9b9: Neither blocks nor log segments found. Creating new log.
I20260928 04:45:42.755288 15361 log.cc:824] T 00000000000000000000000000000000 P 65a485a2db7040fd9f7a0c07ca2ee9b9: Log is configured to *not* fsync() on all Append() calls
I20260928 04:45:42.756114 15361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65a485a2db7040fd9f7a0c07ca2ee9b9: No bootstrap required, opened a new log
I20260928 04:45:42.757741 15361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65a485a2db7040fd9f7a0c07ca2ee9b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec47c0dfd974f658d99c533daa92f85" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38645 } } peers { permanent_uuid: "65a485a2db7040fd9f7a0c07ca2ee9b9" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 43943 } } peers { permanent_uuid: "90afcd51275f414da130bea3e56cd5f4" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45569 } }
I20260928 04:45:42.757892 15361 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65a485a2db7040fd9f7a0c07ca2ee9b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:45:42.757975 15361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65a485a2db7040fd9f7a0c07ca2ee9b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a485a2db7040fd9f7a0c07ca2ee9b9, State: Initialized, Role: FOLLOWER
I20260928 04:45:42.758152 15361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65a485a2db7040fd9f7a0c07ca2ee9b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec47c0dfd974f658d99c533daa92f85" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38645 } } peers { permanent_uuid: "65a485a2db7040fd9f7a0c07ca2ee9b9" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 43943 } } peers { permanent_uuid: "90afcd51275f414da130bea3e56cd5f4" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45569 } }
I20260928 04:45:42.759102 15478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65a485a2db7040fd9f7a0c07ca2ee9b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec47c0dfd974f658d99c533daa92f85" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38645 } } peers { permanent_uuid: "65a485a2db7040fd9f7a0c07ca2ee9b9" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 43943 } } peers { permanent_uuid: "90afcd51275f414da130bea3e56cd5f4" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45569 } } }
I20260928 04:45:42.759176 15361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65a485a2db7040fd9f7a0c07ca2ee9b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:45:42.759249 15478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65a485a2db7040fd9f7a0c07ca2ee9b9 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 04:45:42.761279 15489 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65a485a2db7040fd9f7a0c07ca2ee9b9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:45:42.761368 15489 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:45:42.763715 15293 master_runner.cc:263] No leader master found from master 65a485a2db7040fd9f7a0c07ca2ee9b9
I20260928 04:45:42.763772 15293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 04:45:42.793443 15457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:45:42.793610 15457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec47c0dfd974f658d99c533daa92f85" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38645 } } peers { permanent_uuid: "65a485a2db7040fd9f7a0c07ca2ee9b9" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 43943 } } peers { permanent_uuid: "90afcd51275f414da130bea3e56cd5f4" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45569 } }
I20260928 04:45:42.793885 15457 leader_election.cc:288] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fec47c0dfd974f658d99c533daa92f85 (127.1.226.254:38645), 65a485a2db7040fd9f7a0c07ca2ee9b9 (127.1.226.253:43943)
I20260928 04:45:42.794375 15260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "90afcd51275f414da130bea3e56cd5f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec47c0dfd974f658d99c533daa92f85" is_pre_election: true
I20260928 04:45:42.794553 15260 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fec47c0dfd974f658d99c533daa92f85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90afcd51275f414da130bea3e56cd5f4 in term 0.
I20260928 04:45:42.794786 15384 leader_election.cc:302] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90afcd51275f414da130bea3e56cd5f4, fec47c0dfd974f658d99c533daa92f85; no voters:
I20260928 04:45:42.795329 15335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "90afcd51275f414da130bea3e56cd5f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a485a2db7040fd9f7a0c07ca2ee9b9" is_pre_election: true
I20260928 04:45:42.795531 15335 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65a485a2db7040fd9f7a0c07ca2ee9b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90afcd51275f414da130bea3e56cd5f4 in term 0.
I20260928 04:45:42.795845 15457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:45:42.795902 15457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:45:42.795925 15457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:45:42.797014 15457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec47c0dfd974f658d99c533daa92f85" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38645 } } peers { permanent_uuid: "65a485a2db7040fd9f7a0c07ca2ee9b9" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 43943 } } peers { permanent_uuid: "90afcd51275f414da130bea3e56cd5f4" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45569 } }
I20260928 04:45:42.797148 15457 leader_election.cc:288] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4 [CANDIDATE]: Term 1 election: Requested vote from peers fec47c0dfd974f658d99c533daa92f85 (127.1.226.254:38645), 65a485a2db7040fd9f7a0c07ca2ee9b9 (127.1.226.253:43943)
I20260928 04:45:42.797408 15260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "90afcd51275f414da130bea3e56cd5f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec47c0dfd974f658d99c533daa92f85"
I20260928 04:45:42.797489 15260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fec47c0dfd974f658d99c533daa92f85 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:45:42.798312 15260 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fec47c0dfd974f658d99c533daa92f85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90afcd51275f414da130bea3e56cd5f4 in term 1.
I20260928 04:45:42.798532 15384 leader_election.cc:302] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90afcd51275f414da130bea3e56cd5f4, fec47c0dfd974f658d99c533daa92f85; no voters:
I20260928 04:45:42.798772 15335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "90afcd51275f414da130bea3e56cd5f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a485a2db7040fd9f7a0c07ca2ee9b9"
I20260928 04:45:42.798866 15335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65a485a2db7040fd9f7a0c07ca2ee9b9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:45:42.799727 15335 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65a485a2db7040fd9f7a0c07ca2ee9b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90afcd51275f414da130bea3e56cd5f4 in term 1.
I20260928 04:45:42.800959 15457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:45:42.801137 15457 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4 [term 1 LEADER]: Becoming Leader. State: Replica: 90afcd51275f414da130bea3e56cd5f4, State: Running, Role: LEADER
I20260928 04:45:42.801234 15457 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec47c0dfd974f658d99c533daa92f85" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38645 } } peers { permanent_uuid: "65a485a2db7040fd9f7a0c07ca2ee9b9" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 43943 } } peers { permanent_uuid: "90afcd51275f414da130bea3e56cd5f4" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45569 } }
I20260928 04:45:42.801604 15457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90afcd51275f414da130bea3e56cd5f4. Latest consensus state: current_term: 1 leader_uuid: "90afcd51275f414da130bea3e56cd5f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec47c0dfd974f658d99c533daa92f85" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38645 } } peers { permanent_uuid: "65a485a2db7040fd9f7a0c07ca2ee9b9" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 43943 } } peers { permanent_uuid: "90afcd51275f414da130bea3e56cd5f4" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45569 } } }
I20260928 04:45:42.801666 15457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4 [sys.catalog]: This master's current role is: LEADER
I20260928 04:45:42.802011 15493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:45:42.802374 15493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:45:42.804436 15335 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65a485a2db7040fd9f7a0c07ca2ee9b9 [term 1 FOLLOWER]: Refusing update from remote peer 90afcd51275f414da130bea3e56cd5f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:45:42.804787 15457 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a485a2db7040fd9f7a0c07ca2ee9b9" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 43943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:45:42.806092 15478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65a485a2db7040fd9f7a0c07ca2ee9b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90afcd51275f414da130bea3e56cd5f4. Latest consensus state: current_term: 1 leader_uuid: "90afcd51275f414da130bea3e56cd5f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec47c0dfd974f658d99c533daa92f85" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38645 } } peers { permanent_uuid: "65a485a2db7040fd9f7a0c07ca2ee9b9" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 43943 } } peers { permanent_uuid: "90afcd51275f414da130bea3e56cd5f4" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45569 } } }
I20260928 04:45:42.806183 15478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65a485a2db7040fd9f7a0c07ca2ee9b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:45:42.806623 15260 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fec47c0dfd974f658d99c533daa92f85 [term 1 FOLLOWER]: Refusing update from remote peer 90afcd51275f414da130bea3e56cd5f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:45:42.806913 15457 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec47c0dfd974f658d99c533daa92f85" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:45:42.808017 15443 sys_catalog.cc:455] T 00000000000000000000000000000000 P fec47c0dfd974f658d99c533daa92f85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90afcd51275f414da130bea3e56cd5f4. Latest consensus state: current_term: 1 leader_uuid: "90afcd51275f414da130bea3e56cd5f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec47c0dfd974f658d99c533daa92f85" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38645 } } peers { permanent_uuid: "65a485a2db7040fd9f7a0c07ca2ee9b9" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 43943 } } peers { permanent_uuid: "90afcd51275f414da130bea3e56cd5f4" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45569 } } }
I20260928 04:45:42.808108 15443 sys_catalog.cc:458] T 00000000000000000000000000000000 P fec47c0dfd974f658d99c533daa92f85 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:45:42.809468 15457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "90afcd51275f414da130bea3e56cd5f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec47c0dfd974f658d99c533daa92f85" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38645 } } peers { permanent_uuid: "65a485a2db7040fd9f7a0c07ca2ee9b9" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 43943 } } peers { permanent_uuid: "90afcd51275f414da130bea3e56cd5f4" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45569 } } }
I20260928 04:45:42.809543 15457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4 [sys.catalog]: This master's current role is: LEADER
I20260928 04:45:42.809940 15478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65a485a2db7040fd9f7a0c07ca2ee9b9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "90afcd51275f414da130bea3e56cd5f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec47c0dfd974f658d99c533daa92f85" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38645 } } peers { permanent_uuid: "65a485a2db7040fd9f7a0c07ca2ee9b9" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 43943 } } peers { permanent_uuid: "90afcd51275f414da130bea3e56cd5f4" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45569 } } }
I20260928 04:45:42.810038 15478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65a485a2db7040fd9f7a0c07ca2ee9b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:45:42.810318 15494 mvcc.cc:205] Tried to move back new op lower bound from 7334177762524635136 to 7334177762514317312. Current Snapshot: MvccSnapshot[applied={T|T < 7334177762524635136}]
I20260928 04:45:42.812043 15495 mvcc.cc:205] Tried to move back new op lower bound from 7334177762524635136 to 7334177762514317312. Current Snapshot: MvccSnapshot[applied={T|T < 7334177762524635136}]
I20260928 04:45:42.812628 15493 catalog_manager.cc:1347] Generated new cluster ID: 89abfcc5fe8a4d5cbd7a1a5b1c0ea4bd
I20260928 04:45:42.812680 15493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:45:42.812896 15491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "90afcd51275f414da130bea3e56cd5f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec47c0dfd974f658d99c533daa92f85" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38645 } } peers { permanent_uuid: "65a485a2db7040fd9f7a0c07ca2ee9b9" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 43943 } } peers { permanent_uuid: "90afcd51275f414da130bea3e56cd5f4" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45569 } } }
I20260928 04:45:42.812973 15491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4 [sys.catalog]: This master's current role is: LEADER
I20260928 04:45:42.813369 15443 sys_catalog.cc:455] T 00000000000000000000000000000000 P fec47c0dfd974f658d99c533daa92f85 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "90afcd51275f414da130bea3e56cd5f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec47c0dfd974f658d99c533daa92f85" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38645 } } peers { permanent_uuid: "65a485a2db7040fd9f7a0c07ca2ee9b9" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 43943 } } peers { permanent_uuid: "90afcd51275f414da130bea3e56cd5f4" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 45569 } } }
I20260928 04:45:42.813441 15443 sys_catalog.cc:458] T 00000000000000000000000000000000 P fec47c0dfd974f658d99c533daa92f85 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:45:42.813648 15497 mvcc.cc:205] Tried to move back new op lower bound from 7334177762524635136 to 7334177762514317312. Current Snapshot: MvccSnapshot[applied={T|T < 7334177762524635136}]
I20260928 04:45:42.863040 15493 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:45:42.863931 15493 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:45:42.876611 15493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90afcd51275f414da130bea3e56cd5f4: Generated new TSK 0
I20260928 04:45:42.876969 15493 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:45:43.706625 15454 catalog_manager.cc:1259] Loaded cluster ID: 89abfcc5fe8a4d5cbd7a1a5b1c0ea4bd
I20260928 04:45:43.706698 15454 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fec47c0dfd974f658d99c533daa92f85: loading cluster ID for follower catalog manager: success
I20260928 04:45:43.707509 15454 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fec47c0dfd974f658d99c533daa92f85: acquiring CA information for follower catalog manager: success
I20260928 04:45:43.707813 15454 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fec47c0dfd974f658d99c533daa92f85: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 04:45:43.761807 15489 catalog_manager.cc:1259] Loaded cluster ID: 89abfcc5fe8a4d5cbd7a1a5b1c0ea4bd
I20260928 04:45:43.761922 15489 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 65a485a2db7040fd9f7a0c07ca2ee9b9: loading cluster ID for follower catalog manager: success
I20260928 04:45:43.762768 15489 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 65a485a2db7040fd9f7a0c07ca2ee9b9: acquiring CA information for follower catalog manager: success
I20260928 04:45:43.763130 15489 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 65a485a2db7040fd9f7a0c07ca2ee9b9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 04:45:43.930827 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 15218
I20260928 04:45:43.936404 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 15293
I20260928 04:45:43.941746 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 15368
2026-09-28T04:45:43Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2935 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:45:43 dist-test-slave-j5c3 krb5kdc[15510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:45:43 dist-test-slave-j5c3 krb5kdc[15510](info): set up 2 sockets
Sep 28 04:45:43 dist-test-slave-j5c3 krb5kdc[15510](info): commencing operation
krb5kdc: starting...
W20260928 04:45:44.980386 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:45:45 dist-test-slave-j5c3 krb5kdc[15510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:45:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:45:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:45:45.103317 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:33357
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:43581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:33357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:45:45.185034 15523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:45:45.185307 15523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:45:45.185348 15523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:45:45.186596 15523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:45:45.186672 15523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:45:45.186694 15523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:45:45.186715 15523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:45:45.186753 15523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:45:45.188344 15523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:43581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:33357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:33357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:45:45.188635 15523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:45:45.188879 15523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:45:45.193956 15528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:45.194000 15529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:45.194042 15531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:45.194312 15523 server_base.cc:1034] running on GCE node
I20260928 04:45:45.194530 15523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:45:45.194769 15523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:45:45.195909 15523 hybrid_clock.cc:648] HybridClock initialized: now 1790570745195885 us; error 40 us; skew 500 ppm
Sep 28 04:45:45 dist-test-slave-j5c3 krb5kdc[15510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570745, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:45:45.198553 15523 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:45:45.199055 15523 webserver.cc:460] Webserver started at http://127.1.226.254:37377/ using document root <none> and password file <none>
I20260928 04:45:45.199321 15523 fs_manager.cc:359] Metadata directory not provided
I20260928 04:45:45.199369 15523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:45:45.199520 15523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:45:45.200644 15523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "d21de8118f6449de95152c6d5a163e02"
format_stamp: "Formatted at 2026-09-28 04:45:45 on dist-test-slave-j5c3"
server_key: "6bc1dc8735dd9035547d29283566907e"
server_key_iv: "340818e9d8dfb33c7f2f7858766aa5e0"
server_key_version: "encryptionkey@0"
I20260928 04:45:45.201027 15523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "d21de8118f6449de95152c6d5a163e02"
format_stamp: "Formatted at 2026-09-28 04:45:45 on dist-test-slave-j5c3"
server_key: "6bc1dc8735dd9035547d29283566907e"
server_key_iv: "340818e9d8dfb33c7f2f7858766aa5e0"
server_key_version: "encryptionkey@0"
I20260928 04:45:45.202499 15523 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:45:45.203256 15538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:45.203428 15523 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:45:45.203537 15523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "d21de8118f6449de95152c6d5a163e02"
format_stamp: "Formatted at 2026-09-28 04:45:45 on dist-test-slave-j5c3"
server_key: "6bc1dc8735dd9035547d29283566907e"
server_key_iv: "340818e9d8dfb33c7f2f7858766aa5e0"
server_key_version: "encryptionkey@0"
I20260928 04:45:45.203626 15523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:45:45.215406 15523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:45:45.217612 15523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:45:45.217824 15523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:45:45.222491 15523 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:33357
I20260928 04:45:45.222569 15590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:33357 every 8 connection(s)
I20260928 04:45:45.222927 15523 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:45:45.224043 15591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:45:45.227124 15591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d21de8118f6449de95152c6d5a163e02: Bootstrap starting.
I20260928 04:45:45.227875 15591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d21de8118f6449de95152c6d5a163e02: Neither blocks nor log segments found. Creating new log.
I20260928 04:45:45.228245 15591 log.cc:824] T 00000000000000000000000000000000 P d21de8118f6449de95152c6d5a163e02: Log is configured to *not* fsync() on all Append() calls
I20260928 04:45:45.228418 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 15523
I20260928 04:45:45.228518 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:45:45.228698 1931 external_mini_cluster.cc:1428] Setting key 41ebf6ad1ff7ba1f7e5703021f4cba54
I20260928 04:45:45.229167 15591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d21de8118f6449de95152c6d5a163e02: No bootstrap required, opened a new log
I20260928 04:45:45.231336 15591 raft_consensus.cc:348] T 00000000000000000000000000000000 P d21de8118f6449de95152c6d5a163e02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21de8118f6449de95152c6d5a163e02" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33357 } }
I20260928 04:45:45.231523 15591 raft_consensus.cc:374] T 00000000000000000000000000000000 P d21de8118f6449de95152c6d5a163e02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:45:45.231556 15591 raft_consensus.cc:729] T 00000000000000000000000000000000 P d21de8118f6449de95152c6d5a163e02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d21de8118f6449de95152c6d5a163e02, State: Initialized, Role: FOLLOWER
I20260928 04:45:45.231705 15591 consensus_queue.cc:260] T 00000000000000000000000000000000 P d21de8118f6449de95152c6d5a163e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21de8118f6449de95152c6d5a163e02" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33357 } }
I20260928 04:45:45.231791 15591 raft_consensus.cc:388] T 00000000000000000000000000000000 P d21de8118f6449de95152c6d5a163e02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:45:45.231822 15591 raft_consensus.cc:482] T 00000000000000000000000000000000 P d21de8118f6449de95152c6d5a163e02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:45:45.231870 15591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d21de8118f6449de95152c6d5a163e02 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:45:45.232623 15591 raft_consensus.cc:504] T 00000000000000000000000000000000 P d21de8118f6449de95152c6d5a163e02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21de8118f6449de95152c6d5a163e02" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33357 } }
I20260928 04:45:45.232733 15591 leader_election.cc:302] T 00000000000000000000000000000000 P d21de8118f6449de95152c6d5a163e02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d21de8118f6449de95152c6d5a163e02; no voters:
I20260928 04:45:45.232882 15591 leader_election.cc:288] T 00000000000000000000000000000000 P d21de8118f6449de95152c6d5a163e02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:45:45.232919 15596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d21de8118f6449de95152c6d5a163e02 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:45:45.233103 15596 raft_consensus.cc:686] T 00000000000000000000000000000000 P d21de8118f6449de95152c6d5a163e02 [term 1 LEADER]: Becoming Leader. State: Replica: d21de8118f6449de95152c6d5a163e02, State: Running, Role: LEADER
I20260928 04:45:45.233157 15591 sys_catalog.cc:564] T 00000000000000000000000000000000 P d21de8118f6449de95152c6d5a163e02 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:45:45.233254 15596 consensus_queue.cc:237] T 00000000000000000000000000000000 P d21de8118f6449de95152c6d5a163e02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21de8118f6449de95152c6d5a163e02" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33357 } }
I20260928 04:45:45.235002 15597 sys_catalog.cc:455] T 00000000000000000000000000000000 P d21de8118f6449de95152c6d5a163e02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d21de8118f6449de95152c6d5a163e02. Latest consensus state: current_term: 1 leader_uuid: "d21de8118f6449de95152c6d5a163e02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21de8118f6449de95152c6d5a163e02" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33357 } } }
I20260928 04:45:45.235102 15597 sys_catalog.cc:458] T 00000000000000000000000000000000 P d21de8118f6449de95152c6d5a163e02 [sys.catalog]: This master's current role is: LEADER
I20260928 04:45:45.235399 15598 sys_catalog.cc:455] T 00000000000000000000000000000000 P d21de8118f6449de95152c6d5a163e02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d21de8118f6449de95152c6d5a163e02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21de8118f6449de95152c6d5a163e02" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 33357 } } }
W20260928 04:45:45.235458 15611 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d21de8118f6449de95152c6d5a163e02: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 04:45:45.235843 15598 sys_catalog.cc:458] T 00000000000000000000000000000000 P d21de8118f6449de95152c6d5a163e02 [sys.catalog]: This master's current role is: LEADER
W20260928 04:45:45.235857 15611 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:45:45.237125 15613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:45:45.237385 15613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 04:45:45 dist-test-slave-j5c3 krb5kdc[15510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:45:45.239114 15613 catalog_manager.cc:1347] Generated new cluster ID: 761325fb43de44538d49201ebc7fdb0a
I20260928 04:45:45.239166 15613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:45:45.243140 15595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:45.229366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57700 (local address 127.1.226.254:33357)
0928 04:45:45.229601 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:45.229612 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:45.229983 (+ 371us) server_negotiation.cc:407] Connection header received
0928 04:45:45.230527 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:45.230536 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:45.230600 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:45.230751 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:45.231381 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:45.233101 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:45.234285 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:45.235108 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:45.235266 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:45.238218 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:45.238222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:45.238222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:45.238235 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:45.241411 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:45.241846 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:45.241850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:45.241850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:45.241989 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:45.242198 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:45.242199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:45.242199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:45.242533 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:45.242695 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:45.242781 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 04:45:45.242932 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":112,"threads_started":1}
I20260928 04:45:45.244093 1931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:45:45.247819 15613 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:45:45.248567 15613 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:45:45.266373 15613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d21de8118f6449de95152c6d5a163e02: Generated new TSK 0
I20260928 04:45:45.266623 15613 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:45:45.298177 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 15523
2026-09-28T04:45:45Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1354 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:45:45 dist-test-slave-j5c3 krb5kdc[15618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:45:45 dist-test-slave-j5c3 krb5kdc[15618](info): set up 2 sockets
Sep 28 04:45:45 dist-test-slave-j5c3 krb5kdc[15618](info): commencing operation
krb5kdc: starting...
W20260928 04:45:46.332394 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:45:46 dist-test-slave-j5c3 krb5kdc[15618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:45:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:45:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:45:46.451628 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:36905
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:36905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:45:46.537883 15631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:45:46.538153 15631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:45:46.538196 15631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:45:46.539369 15631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:45:46.539449 15631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:45:46.539479 15631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:45:46.539511 15631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:45:46.539539 15631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:45:46.541127 15631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:36905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:36905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:45:46.541420 15631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:45:46.541682 15631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:45:46.547011 15636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:46.547065 15637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:46.547109 15631 server_base.cc:1034] running on GCE node
W20260928 04:45:46.547036 15639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:46.547524 15631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:45:46.547857 15631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:45:46.549013 15631 hybrid_clock.cc:648] HybridClock initialized: now 1790570746548972 us; error 60 us; skew 500 ppm
Sep 28 04:45:46 dist-test-slave-j5c3 krb5kdc[15618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570746, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:45:46.556023 15631 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:45:46.556641 15631 webserver.cc:460] Webserver started at http://127.1.226.254:40761/ using document root <none> and password file <none>
I20260928 04:45:46.556887 15631 fs_manager.cc:359] Metadata directory not provided
I20260928 04:45:46.556959 15631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:45:46.557123 15631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:45:46.558238 15631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "1a0ae1415dd94dc483884d84a07bcb1c"
format_stamp: "Formatted at 2026-09-28 04:45:46 on dist-test-slave-j5c3"
server_key: "dec1fd3266d6b46b150360f071114e19"
server_key_iv: "fda9c95cbeddf048fe88bab4893da9d9"
server_key_version: "encryptionkey@0"
I20260928 04:45:46.558630 15631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "1a0ae1415dd94dc483884d84a07bcb1c"
format_stamp: "Formatted at 2026-09-28 04:45:46 on dist-test-slave-j5c3"
server_key: "dec1fd3266d6b46b150360f071114e19"
server_key_iv: "fda9c95cbeddf048fe88bab4893da9d9"
server_key_version: "encryptionkey@0"
I20260928 04:45:46.560020 15631 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:45:46.560873 15646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:46.561051 15631 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:45:46.561169 15631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "1a0ae1415dd94dc483884d84a07bcb1c"
format_stamp: "Formatted at 2026-09-28 04:45:46 on dist-test-slave-j5c3"
server_key: "dec1fd3266d6b46b150360f071114e19"
server_key_iv: "fda9c95cbeddf048fe88bab4893da9d9"
server_key_version: "encryptionkey@0"
I20260928 04:45:46.561246 15631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:45:46.579483 15631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:45:46.581823 15631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:45:46.581998 15631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:45:46.586361 15631 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:36905
I20260928 04:45:46.586380 15698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:36905 every 8 connection(s)
I20260928 04:45:46.586773 15631 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:45:46.586920 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 15631
I20260928 04:45:46.587067 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:45:46.587294 1931 external_mini_cluster.cc:1428] Setting key f4ebd7184cfc9e413f294ada5b3b6433
I20260928 04:45:46.588030 15699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:45:46.591039 15699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c: Bootstrap starting.
I20260928 04:45:46.591739 15699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c: Neither blocks nor log segments found. Creating new log.
I20260928 04:45:46.592052 15699 log.cc:824] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c: Log is configured to *not* fsync() on all Append() calls
I20260928 04:45:46.593000 15699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c: No bootstrap required, opened a new log
Sep 28 04:45:46 dist-test-slave-j5c3 krb5kdc[15618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:45:46.595196 15699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0ae1415dd94dc483884d84a07bcb1c" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 36905 } }
I20260928 04:45:46.595369 15699 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:45:46.595418 15699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a0ae1415dd94dc483884d84a07bcb1c, State: Initialized, Role: FOLLOWER
I20260928 04:45:46.595594 15699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0ae1415dd94dc483884d84a07bcb1c" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 36905 } }
I20260928 04:45:46.595692 15699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:45:46.595736 15699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:45:46.595785 15699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:45:46.596436 15699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0ae1415dd94dc483884d84a07bcb1c" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 36905 } }
I20260928 04:45:46.596560 15699 leader_election.cc:302] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a0ae1415dd94dc483884d84a07bcb1c; no voters:
I20260928 04:45:46.596778 15699 leader_election.cc:288] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:45:46.596812 15704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:45:46.596992 15704 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [term 1 LEADER]: Becoming Leader. State: Replica: 1a0ae1415dd94dc483884d84a07bcb1c, State: Running, Role: LEADER
I20260928 04:45:46.597064 15699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:45:46.597137 15704 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0ae1415dd94dc483884d84a07bcb1c" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 36905 } }
I20260928 04:45:46.597560 15706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a0ae1415dd94dc483884d84a07bcb1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0ae1415dd94dc483884d84a07bcb1c" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 36905 } } }
I20260928 04:45:46.597751 15705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a0ae1415dd94dc483884d84a07bcb1c. Latest consensus state: current_term: 1 leader_uuid: "1a0ae1415dd94dc483884d84a07bcb1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0ae1415dd94dc483884d84a07bcb1c" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 36905 } } }
I20260928 04:45:46.597898 15705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [sys.catalog]: This master's current role is: LEADER
I20260928 04:45:46.598485 15715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:45:46.597709 15706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [sys.catalog]: This master's current role is: LEADER
I20260928 04:45:46.598888 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:46.587856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36386 (local address 127.1.226.254:36905)
0928 04:45:46.588072 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:46.588080 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:46.588106 (+ 26us) server_negotiation.cc:407] Connection header received
0928 04:45:46.588600 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:46.588606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:46.588661 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:46.588784 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:46.589593 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:46.590529 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:46.591598 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:46.592052 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:46.592156 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:46.594900 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:46.594904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:46.594904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:46.594915 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:46.596997 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:46.597383 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:46.597386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:46.597386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:46.597487 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:46.597711 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:46.597712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:46.597712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:46.598089 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:46.598384 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:46.598476 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 04:45:46.598637 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":120,"threads_started":1}
I20260928 04:45:46.599068 15715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:45:46.600602 15715 catalog_manager.cc:1347] Generated new cluster ID: c8e908171bd24a1ca57f81a48a3d9fb5
I20260928 04:45:46.600674 15715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:45:46.611285 15715 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:45:46.611795 15715 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:45:46.617465 15715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c: Generated new TSK 0
I20260928 04:45:46.617664 15715 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.226.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
I20260928 04:45:46.649214 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:0
--local_ip_for_outbound_sockets=127.1.226.193
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:36905
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:45:46.734272 15725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:45:46.734548 15725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:45:46.734597 15725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:45:46.735867 15725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:45:46.735947 15725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:45:46.735994 15725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:45:46.737632 15725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:36905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:45:46.737949 15725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:45:46.738219 15725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:45:46.743609 15730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:46.743609 15731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:46.743641 15733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:46.744073 15725 server_base.cc:1034] running on GCE node
I20260928 04:45:46.744432 15725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:45:46.744676 15725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:45:46.745855 15725 hybrid_clock.cc:648] HybridClock initialized: now 1790570746745839 us; error 29 us; skew 500 ppm
Sep 28 04:45:46 dist-test-slave-j5c3 krb5kdc[15618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570746, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:45:46.748661 15725 init.cc:377] Logged in from keytab as kudu/127.1.226.193@KRBTEST.COM (short username kudu)
I20260928 04:45:46.749171 15725 webserver.cc:460] Webserver started at http://127.1.226.193:38635/ using document root <none> and password file <none>
I20260928 04:45:46.749452 15725 fs_manager.cc:359] Metadata directory not provided
I20260928 04:45:46.749528 15725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:45:46.749680 15725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:45:46.750834 15725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/data/instance:
uuid: "3cfb71b57e6648cbb0786b6250f3e5c9"
format_stamp: "Formatted at 2026-09-28 04:45:46 on dist-test-slave-j5c3"
server_key: "0833c0510044e057eb27d0b290dd40e8"
server_key_iv: "263e9b97f9e55a381c914d87c669cb6b"
server_key_version: "encryptionkey@0"
I20260928 04:45:46.751204 15725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance:
uuid: "3cfb71b57e6648cbb0786b6250f3e5c9"
format_stamp: "Formatted at 2026-09-28 04:45:46 on dist-test-slave-j5c3"
server_key: "0833c0510044e057eb27d0b290dd40e8"
server_key_iv: "263e9b97f9e55a381c914d87c669cb6b"
server_key_version: "encryptionkey@0"
I20260928 04:45:46.752776 15725 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:45:46.753760 15740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:46.753926 15725 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:45:46.754040 15725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "3cfb71b57e6648cbb0786b6250f3e5c9"
format_stamp: "Formatted at 2026-09-28 04:45:46 on dist-test-slave-j5c3"
server_key: "0833c0510044e057eb27d0b290dd40e8"
server_key_iv: "263e9b97f9e55a381c914d87c669cb6b"
server_key_version: "encryptionkey@0"
I20260928 04:45:46.754122 15725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:45:46.768471 15725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:45:46.770982 15725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:45:46.771160 15725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:45:46.771440 15725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:45:46.771756 15725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:45:46.771811 15725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:46.771852 15725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:45:46.771881 15725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:46.777992 15725 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:43663
I20260928 04:45:46.778083 15853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:43663 every 8 connection(s)
I20260928 04:45:46.778474 15725 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
Sep 28 04:45:46 dist-test-slave-j5c3 krb5kdc[15618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570746, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:45:46.784957 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 15725
I20260928 04:45:46.785070 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance
I20260928 04:45:46.785275 1931 external_mini_cluster.cc:1428] Setting key 2219ea7b2a6eca7dc10dfa98baf76ac2
I20260928 04:45:46.788715 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:46.779355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:55683 (local address 127.1.226.254:36905)
0928 04:45:46.779441 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:46.779443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:46.779623 (+ 180us) server_negotiation.cc:407] Connection header received
0928 04:45:46.780127 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:46.780128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:46.780138 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:46.780263 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:46.781214 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:46.781759 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:46.782764 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:46.782965 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:46.783023 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:46.785324 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:46.785326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:46.785326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:46.785335 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:46.786982 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:46.787527 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:46.787528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:46.787528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:46.787599 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:46.787895 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:46.787896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:46.787897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:46.788078 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:46.788172 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:46.788481 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 04:45:46.788573 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 04:45:46.788739 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:46.779298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.193:55683)
0928 04:45:46.779492 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:46.779534 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:46.780041 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:46.780279 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:46.780283 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:46.780890 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:45:46.781094 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:46.781099 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:46.781878 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:46.781879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:46.782681 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:46.782682 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:46.783046 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:46.783047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:46.783159 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:46.783679 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:45:46.783688 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:45:46.785216 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:45:46.787067 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:46.787077 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:46.787080 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:46.787437 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:46.787670 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:46.787671 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:46.787671 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:46.787826 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:46.788148 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:45:46.788189 (+ 41us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:45:46.788339 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 04:45:46.788430 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 04:45:46.788563 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":98,"threads_started":1}
I20260928 04:45:46.789342 15854 heartbeater.cc:347] Connected to a master server at 127.1.226.254:36905
I20260928 04:45:46.789463 15854 heartbeater.cc:464] Registering TS with master...
I20260928 04:45:46.789672 15854 heartbeater.cc:511] Master 127.1.226.254:36905 requested a full tablet report, sending...
I20260928 04:45:46.790243 15663 ts_manager.cc:194] Registered new tserver with Master: 3cfb71b57e6648cbb0786b6250f3e5c9 (127.1.226.193:43663)
I20260928 04:45:46.790932 15663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.193@KRBTEST.COM'} at 127.1.226.193:55683
WARNING: no policy specified for kudu/127.1.226.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
I20260928 04:45:46.811896 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:0
--local_ip_for_outbound_sockets=127.1.226.194
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:36905
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:45:46.894398 15859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:45:46.894629 15859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:45:46.894656 15859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:45:46.895892 15859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:45:46.895941 15859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:45:46.895969 15859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:45:46.897601 15859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:36905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:45:46.897927 15859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:45:46.898190 15859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:45:46.903556 15867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:46.903538 15865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:46.903538 15864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:46.903582 15859 server_base.cc:1034] running on GCE node
I20260928 04:45:46.904055 15859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:45:46.904323 15859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:45:46.905503 15859 hybrid_clock.cc:648] HybridClock initialized: now 1790570746905495 us; error 38 us; skew 500 ppm
Sep 28 04:45:46 dist-test-slave-j5c3 krb5kdc[15618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570746, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:45:46.908246 15859 init.cc:377] Logged in from keytab as kudu/127.1.226.194@KRBTEST.COM (short username kudu)
I20260928 04:45:46.908695 15859 webserver.cc:460] Webserver started at http://127.1.226.194:42245/ using document root <none> and password file <none>
I20260928 04:45:46.908931 15859 fs_manager.cc:359] Metadata directory not provided
I20260928 04:45:46.908996 15859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:45:46.909147 15859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:45:46.910250 15859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/data/instance:
uuid: "bdbad0e08ce14c83ab0de7783bdddd09"
format_stamp: "Formatted at 2026-09-28 04:45:46 on dist-test-slave-j5c3"
server_key: "9c539262f6d69ee1539b64df0cb6e0fc"
server_key_iv: "58630404ba18b4b6605f8146f62c191e"
server_key_version: "encryptionkey@0"
I20260928 04:45:46.910619 15859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance:
uuid: "bdbad0e08ce14c83ab0de7783bdddd09"
format_stamp: "Formatted at 2026-09-28 04:45:46 on dist-test-slave-j5c3"
server_key: "9c539262f6d69ee1539b64df0cb6e0fc"
server_key_iv: "58630404ba18b4b6605f8146f62c191e"
server_key_version: "encryptionkey@0"
I20260928 04:45:46.911957 15859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:45:46.912791 15874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:46.912976 15859 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:45:46.913092 15859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "bdbad0e08ce14c83ab0de7783bdddd09"
format_stamp: "Formatted at 2026-09-28 04:45:46 on dist-test-slave-j5c3"
server_key: "9c539262f6d69ee1539b64df0cb6e0fc"
server_key_iv: "58630404ba18b4b6605f8146f62c191e"
server_key_version: "encryptionkey@0"
I20260928 04:45:46.913182 15859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:45:46.937335 15859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:45:46.939499 15859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:45:46.939694 15859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:45:46.939992 15859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:45:46.940351 15859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:45:46.940407 15859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:46.940446 15859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:45:46.940476 15859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:46.946568 15859 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:32989
I20260928 04:45:46.946610 15987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:32989 every 8 connection(s)
I20260928 04:45:46.946965 15859 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
I20260928 04:45:46.947338 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 15859
I20260928 04:45:46.947456 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance
I20260928 04:45:46.947666 1931 external_mini_cluster.cc:1428] Setting key b679b848dcfcb4cb79b14ef5269ccad6
Sep 28 04:45:46 dist-test-slave-j5c3 krb5kdc[15618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570746, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.226.195@KRBTEST.COM; defaulting to no policy
I20260928 04:45:46.957752 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:46.947811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:58945 (local address 127.1.226.254:36905)
0928 04:45:46.947878 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:46.947880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:46.948133 (+ 253us) server_negotiation.cc:407] Connection header received
0928 04:45:46.948797 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:46.948798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:46.948811 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:46.948855 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:46.949942 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:46.950454 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:46.951507 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:46.951803 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:46.951882 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:46.954417 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:46.954419 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:46.954419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:46.954427 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:46.956218 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:46.956701 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:46.956702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:46.956702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:46.956769 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:46.957058 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:46.957058 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:46.957059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:46.957230 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:46.957286 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:46.957600 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 04:45:46.957682 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 04:45:46.957989 15990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:46.947797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.194:58945)
0928 04:45:46.947982 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:46.948036 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:46.948684 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:46.948946 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:46.948950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:46.949623 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:45:46.949841 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:46.949846 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:46.950563 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:46.950564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:46.951397 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:46.951399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:46.951909 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:46.951911 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:46.952047 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:46.952670 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:45:46.952679 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:45:46.954299 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:45:46.956309 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:46.956317 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:46.956319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:46.956556 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:46.956880 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:46.956881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:46.956881 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:46.956997 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:46.957297 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:45:46.957302 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:45:46.957437 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 04:45:46.957567 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 04:45:46.957749 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":95,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.195@KRBTEST.COM" created.
I20260928 04:45:46.958537 15988 heartbeater.cc:347] Connected to a master server at 127.1.226.254:36905
I20260928 04:45:46.958659 15988 heartbeater.cc:464] Registering TS with master...
I20260928 04:45:46.958896 15988 heartbeater.cc:511] Master 127.1.226.254:36905 requested a full tablet report, sending...
I20260928 04:45:46.959334 15663 ts_manager.cc:194] Registered new tserver with Master: bdbad0e08ce14c83ab0de7783bdddd09 (127.1.226.194:32989)
I20260928 04:45:46.959874 15663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.194@KRBTEST.COM'} at 127.1.226.194:58945
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
I20260928 04:45:46.974354 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:0
--local_ip_for_outbound_sockets=127.1.226.195
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:36905
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:45:47.056931 15993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:45:47.057185 15993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:45:47.057220 15993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:45:47.058490 15993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:45:47.058531 15993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:45:47.058558 15993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:45:47.060010 15993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:36905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:45:47.060290 15993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:45:47.060539 15993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:45:47.065639 15998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:47.065620 16001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:47.065610 15999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:47.065855 15993 server_base.cc:1034] running on GCE node
I20260928 04:45:47.066041 15993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:45:47.066236 15993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:45:47.067390 15993 hybrid_clock.cc:648] HybridClock initialized: now 1790570747067375 us; error 29 us; skew 500 ppm
Sep 28 04:45:47 dist-test-slave-j5c3 krb5kdc[15618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570747, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:45:47.070032 15993 init.cc:377] Logged in from keytab as kudu/127.1.226.195@KRBTEST.COM (short username kudu)
I20260928 04:45:47.070488 15993 webserver.cc:460] Webserver started at http://127.1.226.195:44139/ using document root <none> and password file <none>
I20260928 04:45:47.070734 15993 fs_manager.cc:359] Metadata directory not provided
I20260928 04:45:47.070781 15993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:45:47.070952 15993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:45:47.071980 15993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/data/instance:
uuid: "e6d19702051c4ea8bc3c0319416ef50e"
format_stamp: "Formatted at 2026-09-28 04:45:47 on dist-test-slave-j5c3"
server_key: "48bb458b125d07faccd8ca28feb9d2e2"
server_key_iv: "0c219606240e6d95f82d8667ade36b6e"
server_key_version: "encryptionkey@0"
I20260928 04:45:47.072415 15993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance:
uuid: "e6d19702051c4ea8bc3c0319416ef50e"
format_stamp: "Formatted at 2026-09-28 04:45:47 on dist-test-slave-j5c3"
server_key: "48bb458b125d07faccd8ca28feb9d2e2"
server_key_iv: "0c219606240e6d95f82d8667ade36b6e"
server_key_version: "encryptionkey@0"
I20260928 04:45:47.073679 15993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:45:47.074374 16008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:47.074548 15993 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:45:47.074647 15993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "e6d19702051c4ea8bc3c0319416ef50e"
format_stamp: "Formatted at 2026-09-28 04:45:47 on dist-test-slave-j5c3"
server_key: "48bb458b125d07faccd8ca28feb9d2e2"
server_key_iv: "0c219606240e6d95f82d8667ade36b6e"
server_key_version: "encryptionkey@0"
I20260928 04:45:47.074728 15993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:45:47.099273 15993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:45:47.101449 15993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:45:47.101619 15993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:45:47.101862 15993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:45:47.102180 15993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:45:47.102236 15993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:47.102326 15993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:45:47.102357 15993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:47.108286 15993 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:39013
I20260928 04:45:47.108393 16121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:39013 every 8 connection(s)
I20260928 04:45:47.108711 15993 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
I20260928 04:45:47.109549 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 15993
I20260928 04:45:47.109655 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance
I20260928 04:45:47.109830 1931 external_mini_cluster.cc:1428] Setting key 62916fa138772dd0e6f2e002d493f8c8
Sep 28 04:45:47 dist-test-slave-j5c3 krb5kdc[15618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570747, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:45:47.118386 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.109505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:51971 (local address 127.1.226.254:36905)
0928 04:45:47.109576 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:47.109578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:47.109734 (+ 156us) server_negotiation.cc:407] Connection header received
0928 04:45:47.110259 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:47.110260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:47.110273 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:47.110324 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:47.111269 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.111789 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.112897 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.113107 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.113183 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.115469 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:47.115471 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:47.115471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:47.115479 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:47.116932 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:47.117403 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:47.117404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:47.117404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:47.117461 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:47.117798 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:47.117799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:47.117799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:47.117913 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:47.117976 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:47.118248 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 04:45:47.118321 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 04:45:47.118570 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.109460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.195:51971)
0928 04:45:47.109616 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:47.109658 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:47.110148 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:47.110399 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:47.110404 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:47.110968 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:45:47.111168 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:47.111173 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.111894 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.111895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:47.112789 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:47.112792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.113204 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.113205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:47.113348 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.113889 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:45:47.113898 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:45:47.115376 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:45:47.117007 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:47.117013 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:47.117015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:47.117314 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:47.117538 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:47.117539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:47.117539 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:47.117700 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:47.117990 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:45:47.117996 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:45:47.118133 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 04:45:47.118223 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 04:45:47.118354 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260928 04:45:47.119130 16122 heartbeater.cc:347] Connected to a master server at 127.1.226.254:36905
I20260928 04:45:47.119369 16122 heartbeater.cc:464] Registering TS with master...
I20260928 04:45:47.119572 16122 heartbeater.cc:511] Master 127.1.226.254:36905 requested a full tablet report, sending...
I20260928 04:45:47.120015 15663 ts_manager.cc:194] Registered new tserver with Master: e6d19702051c4ea8bc3c0319416ef50e (127.1.226.195:39013)
I20260928 04:45:47.120450 15663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.195@KRBTEST.COM'} at 127.1.226.195:51971
I20260928 04:45:47.121446 1931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:45:47.128820 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.122657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36388 (local address 127.1.226.254:36905)
0928 04:45:47.122706 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:47.122708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:47.122777 (+ 69us) server_negotiation.cc:407] Connection header received
0928 04:45:47.122856 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:47.122857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:47.122866 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:47.122916 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:47.123478 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.123959 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.124773 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.124972 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.125050 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.126187 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:47.126189 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:47.126189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:47.126196 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:47.127316 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:47.127912 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:47.127913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:47.127913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:47.127982 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:47.128244 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:47.128245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:47.128245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:47.128410 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:47.128533 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:47.128636 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 04:45:47.128716 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 04:45:47.130455 15663 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:45:47.132067 15663 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:45:47.138748 16136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.134631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57228 (local address 127.1.226.195:39013)
0928 04:45:47.134853 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:47.134858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:47.134869 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:45:47.134911 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:47.134914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:47.134932 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:47.135221 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:45:47.135307 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.136475 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.137633 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.138346 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.138425 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.138483 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:47.138618 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 04:45:47.138666 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":85,"threads_started":1}
I20260928 04:45:47.138736 16135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.134543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:39013 (local address 127.0.0.1:57228)
0928 04:45:47.134668 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:47.134680 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:47.134741 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:47.135054 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:47.135057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:47.135060 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:45:47.135141 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:47.135143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.136556 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.136558 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:47.137544 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:47.137546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.138431 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.138432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:47.138490 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.138502 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 04:45:47.138593 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 04:45:47.138650 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":6,"thread_start_us":77,"threads_started":1}
I20260928 04:45:47.138859 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.134320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:43663 (local address 127.0.0.1:49400)
0928 04:45:47.134442 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:47.134483 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:47.134543 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:47.135075 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:47.135076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:47.135077 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:45:47.135141 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:47.135144 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.136477 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.136478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:47.137744 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:47.137746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.138578 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.138579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:47.138634 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.138636 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:45:47.138685 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 04:45:47.138720 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":67,"thread_start_us":74,"threads_started":1}
I20260928 04:45:47.139081 16134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.134445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49400 (local address 127.1.226.193:43663)
0928 04:45:47.134918 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:47.134921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:47.134931 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:45:47.134967 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:47.134969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:47.134983 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:47.135219 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:45:47.135294 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.136395 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.137838 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.138501 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.138589 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.138641 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:47.138931 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 04:45:47.138988 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":118,"threads_started":1}
I20260928 04:45:47.139743 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.134733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:32989 (local address 127.0.0.1:44348)
0928 04:45:47.135034 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:47.135043 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:47.135103 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:47.135393 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:47.135394 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:47.135395 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:45:47.135448 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:47.135449 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.136407 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.136408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:47.137744 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:47.137746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.139487 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.139488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:47.139570 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.139576 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 04:45:47.139624 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 04:45:47.139667 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":82,"thread_start_us":66,"threads_started":1}
I20260928 04:45:47.139750 15788 tablet_service.cc:1467] Processing CreateTablet for tablet 49e7f32b8db444088bf73aabd8cce382 (DEFAULT_TABLE table=test-table [id=70361d1b546647c2b4089ce2fdd7e585]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:45:47.140018 16138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.134853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44348 (local address 127.1.226.194:32989)
0928 04:45:47.135015 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:47.135019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:47.135164 (+ 145us) server_negotiation.cc:407] Connection header received
0928 04:45:47.135210 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:47.135212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:47.135227 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:47.135301 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:45:47.135532 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.136304 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.138677 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.139322 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.139397 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.139454 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:47.139885 (+ 431us) server_negotiation.cc:299] Negotiation successful
0928 04:45:47.139930 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20260928 04:45:47.140204 15788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49e7f32b8db444088bf73aabd8cce382. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:45:47.140650 15922 tablet_service.cc:1467] Processing CreateTablet for tablet 49e7f32b8db444088bf73aabd8cce382 (DEFAULT_TABLE table=test-table [id=70361d1b546647c2b4089ce2fdd7e585]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:45:47.140961 15922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49e7f32b8db444088bf73aabd8cce382. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:45:47.143153 16139 tablet_bootstrap.cc:492] T 49e7f32b8db444088bf73aabd8cce382 P 3cfb71b57e6648cbb0786b6250f3e5c9: Bootstrap starting.
I20260928 04:45:47.143671 16140 tablet_bootstrap.cc:492] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09: Bootstrap starting.
I20260928 04:45:47.143584 16056 tablet_service.cc:1467] Processing CreateTablet for tablet 49e7f32b8db444088bf73aabd8cce382 (DEFAULT_TABLE table=test-table [id=70361d1b546647c2b4089ce2fdd7e585]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:45:47.143949 16056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49e7f32b8db444088bf73aabd8cce382. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:45:47.144011 16139 tablet_bootstrap.cc:654] T 49e7f32b8db444088bf73aabd8cce382 P 3cfb71b57e6648cbb0786b6250f3e5c9: Neither blocks nor log segments found. Creating new log.
I20260928 04:45:47.144479 16140 tablet_bootstrap.cc:654] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09: Neither blocks nor log segments found. Creating new log.
I20260928 04:45:47.144621 16139 log.cc:824] T 49e7f32b8db444088bf73aabd8cce382 P 3cfb71b57e6648cbb0786b6250f3e5c9: Log is configured to *not* fsync() on all Append() calls
I20260928 04:45:47.145076 16140 log.cc:824] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09: Log is configured to *not* fsync() on all Append() calls
I20260928 04:45:47.145493 16139 tablet_bootstrap.cc:492] T 49e7f32b8db444088bf73aabd8cce382 P 3cfb71b57e6648cbb0786b6250f3e5c9: No bootstrap required, opened a new log
I20260928 04:45:47.145572 16139 ts_tablet_manager.cc:1397] T 49e7f32b8db444088bf73aabd8cce382 P 3cfb71b57e6648cbb0786b6250f3e5c9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 04:45:47.146013 16140 tablet_bootstrap.cc:492] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09: No bootstrap required, opened a new log
I20260928 04:45:47.146083 16140 ts_tablet_manager.cc:1397] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 04:45:47.147368 16143 tablet_bootstrap.cc:492] T 49e7f32b8db444088bf73aabd8cce382 P e6d19702051c4ea8bc3c0319416ef50e: Bootstrap starting.
I20260928 04:45:47.147696 16139 raft_consensus.cc:348] T 49e7f32b8db444088bf73aabd8cce382 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 } } peers { permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 } } peers { permanent_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32989 } }
I20260928 04:45:47.147629 16140 raft_consensus.cc:348] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 } } peers { permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 } } peers { permanent_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32989 } }
I20260928 04:45:47.147867 16140 raft_consensus.cc:374] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:45:47.147868 16139 raft_consensus.cc:374] T 49e7f32b8db444088bf73aabd8cce382 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:45:47.147913 16139 raft_consensus.cc:729] T 49e7f32b8db444088bf73aabd8cce382 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cfb71b57e6648cbb0786b6250f3e5c9, State: Initialized, Role: FOLLOWER
I20260928 04:45:47.147913 16140 raft_consensus.cc:729] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdbad0e08ce14c83ab0de7783bdddd09, State: Initialized, Role: FOLLOWER
I20260928 04:45:47.148034 16139 consensus_queue.cc:260] T 49e7f32b8db444088bf73aabd8cce382 P 3cfb71b57e6648cbb0786b6250f3e5c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 } } peers { permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 } } peers { permanent_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32989 } }
I20260928 04:45:47.148054 16140 consensus_queue.cc:260] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 } } peers { permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 } } peers { permanent_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32989 } }
I20260928 04:45:47.148228 16143 tablet_bootstrap.cc:654] T 49e7f32b8db444088bf73aabd8cce382 P e6d19702051c4ea8bc3c0319416ef50e: Neither blocks nor log segments found. Creating new log.
I20260928 04:45:47.148393 16140 ts_tablet_manager.cc:1428] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:45:47.148412 16139 ts_tablet_manager.cc:1428] T 49e7f32b8db444088bf73aabd8cce382 P 3cfb71b57e6648cbb0786b6250f3e5c9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 04:45:47.148555 15988 heartbeater.cc:503] Master 127.1.226.254:36905 was elected leader, sending a full tablet report...
I20260928 04:45:47.148829 16143 log.cc:824] T 49e7f32b8db444088bf73aabd8cce382 P e6d19702051c4ea8bc3c0319416ef50e: Log is configured to *not* fsync() on all Append() calls
I20260928 04:45:47.148874 15854 heartbeater.cc:503] Master 127.1.226.254:36905 was elected leader, sending a full tablet report...
I20260928 04:45:47.149641 16143 tablet_bootstrap.cc:492] T 49e7f32b8db444088bf73aabd8cce382 P e6d19702051c4ea8bc3c0319416ef50e: No bootstrap required, opened a new log
I20260928 04:45:47.149761 16143 ts_tablet_manager.cc:1397] T 49e7f32b8db444088bf73aabd8cce382 P e6d19702051c4ea8bc3c0319416ef50e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 04:45:47.151335 16143 raft_consensus.cc:348] T 49e7f32b8db444088bf73aabd8cce382 P e6d19702051c4ea8bc3c0319416ef50e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 } } peers { permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 } } peers { permanent_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32989 } }
I20260928 04:45:47.151513 16143 raft_consensus.cc:374] T 49e7f32b8db444088bf73aabd8cce382 P e6d19702051c4ea8bc3c0319416ef50e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:45:47.151561 16143 raft_consensus.cc:729] T 49e7f32b8db444088bf73aabd8cce382 P e6d19702051c4ea8bc3c0319416ef50e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6d19702051c4ea8bc3c0319416ef50e, State: Initialized, Role: FOLLOWER
I20260928 04:45:47.151690 16143 consensus_queue.cc:260] T 49e7f32b8db444088bf73aabd8cce382 P e6d19702051c4ea8bc3c0319416ef50e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 } } peers { permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 } } peers { permanent_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32989 } }
I20260928 04:45:47.151965 16143 ts_tablet_manager.cc:1428] T 49e7f32b8db444088bf73aabd8cce382 P e6d19702051c4ea8bc3c0319416ef50e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 04:45:47.152034 16122 heartbeater.cc:503] Master 127.1.226.254:36905 was elected leader, sending a full tablet report...
W20260928 04:45:47.197602 15989 tablet.cc:2369] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:45:47.236608 16144 raft_consensus.cc:482] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:45:47.236780 16144 raft_consensus.cc:504] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 } } peers { permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 } } peers { permanent_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32989 } }
I20260928 04:45:47.237145 16144 leader_election.cc:288] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cfb71b57e6648cbb0786b6250f3e5c9 (127.1.226.193:43663), e6d19702051c4ea8bc3c0319416ef50e (127.1.226.195:39013)
I20260928 04:45:47.240098 15990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.237361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:43663 (local address 127.1.226.194:59703)
0928 04:45:47.237429 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:47.237442 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:47.237492 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:47.237726 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:47.237727 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:47.237729 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:45:47.237799 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:47.237800 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.238479 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.238480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:47.239248 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:47.239250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.239822 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.239823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:47.239909 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.239913 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:45:47.239981 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 04:45:47.240025 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":1}
I20260928 04:45:47.240098 16134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.237428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:59703 (local address 127.1.226.193:43663)
0928 04:45:47.237476 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:47.237483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:47.237510 (+ 27us) server_negotiation.cc:407] Connection header received
0928 04:45:47.237567 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:47.237568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:47.237580 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:47.237674 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:45:47.237889 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.238376 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.239346 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.239724 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.239822 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.239831 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:47.239989 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 04:45:47.240025 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:45:47.240272 16136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.237597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:42333 (local address 127.1.226.195:39013)
0928 04:45:47.237675 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:47.237677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:47.237820 (+ 143us) server_negotiation.cc:407] Connection header received
0928 04:45:47.237875 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:47.237876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:47.237885 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:47.237939 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:45:47.238090 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.238559 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.239480 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.239863 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.239960 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.239968 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:47.240120 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 04:45:47.240183 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 04:45:47.240401 16148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.237517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:39013 (local address 127.1.226.194:42333)
0928 04:45:47.237742 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:47.237753 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:47.237814 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:47.237944 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:47.237944 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:47.237946 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:45:47.238014 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:47.238016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.238646 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.238647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:47.239384 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:47.239386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.239962 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.239963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:47.240036 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.240044 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 04:45:47.240103 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 04:45:47.240144 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":2,"thread_start_us":146,"threads_started":1}
I20260928 04:45:47.240483 15808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49e7f32b8db444088bf73aabd8cce382" candidate_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" is_pre_election: true
I20260928 04:45:47.240665 15808 raft_consensus.cc:2454] T 49e7f32b8db444088bf73aabd8cce382 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdbad0e08ce14c83ab0de7783bdddd09 in term 0.
I20260928 04:45:47.240797 16076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49e7f32b8db444088bf73aabd8cce382" candidate_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6d19702051c4ea8bc3c0319416ef50e" is_pre_election: true
I20260928 04:45:47.240926 15875 leader_election.cc:302] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cfb71b57e6648cbb0786b6250f3e5c9, bdbad0e08ce14c83ab0de7783bdddd09; no voters:
I20260928 04:45:47.240967 16076 raft_consensus.cc:2454] T 49e7f32b8db444088bf73aabd8cce382 P e6d19702051c4ea8bc3c0319416ef50e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdbad0e08ce14c83ab0de7783bdddd09 in term 0.
I20260928 04:45:47.241044 16144 raft_consensus.cc:2781] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:45:47.241102 16144 raft_consensus.cc:482] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:45:47.241127 16144 raft_consensus.cc:3037] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:45:47.241837 16144 raft_consensus.cc:504] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 } } peers { permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 } } peers { permanent_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32989 } }
I20260928 04:45:47.241972 16144 leader_election.cc:288] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09 [CANDIDATE]: Term 1 election: Requested vote from peers 3cfb71b57e6648cbb0786b6250f3e5c9 (127.1.226.193:43663), e6d19702051c4ea8bc3c0319416ef50e (127.1.226.195:39013)
I20260928 04:45:47.242168 16076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49e7f32b8db444088bf73aabd8cce382" candidate_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6d19702051c4ea8bc3c0319416ef50e"
I20260928 04:45:47.242240 16076 raft_consensus.cc:3037] T 49e7f32b8db444088bf73aabd8cce382 P e6d19702051c4ea8bc3c0319416ef50e [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:45:47.242961 16076 raft_consensus.cc:2454] T 49e7f32b8db444088bf73aabd8cce382 P e6d19702051c4ea8bc3c0319416ef50e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdbad0e08ce14c83ab0de7783bdddd09 in term 1.
I20260928 04:45:47.243088 15808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49e7f32b8db444088bf73aabd8cce382" candidate_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9"
I20260928 04:45:47.243129 15875 leader_election.cc:302] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdbad0e08ce14c83ab0de7783bdddd09, e6d19702051c4ea8bc3c0319416ef50e; no voters:
I20260928 04:45:47.243194 15808 raft_consensus.cc:3037] T 49e7f32b8db444088bf73aabd8cce382 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:45:47.243232 16144 raft_consensus.cc:2781] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:45:47.243392 16144 raft_consensus.cc:686] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09 [term 1 LEADER]: Becoming Leader. State: Replica: bdbad0e08ce14c83ab0de7783bdddd09, State: Running, Role: LEADER
I20260928 04:45:47.243511 16144 consensus_queue.cc:237] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 } } peers { permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 } } peers { permanent_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32989 } }
I20260928 04:45:47.244103 15808 raft_consensus.cc:2454] T 49e7f32b8db444088bf73aabd8cce382 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdbad0e08ce14c83ab0de7783bdddd09 in term 1.
I20260928 04:45:47.244534 15662 catalog_manager.cc:5495] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09 reported cstate change: term changed from 0 to 1, leader changed from <none> to bdbad0e08ce14c83ab0de7783bdddd09 (127.1.226.194). New cstate: current_term: 1 leader_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32989 } health_report { overall_health: HEALTHY } } }
I20260928 04:45:47.253948 16138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.251671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44352 (local address 127.1.226.194:32989)
0928 04:45:47.251724 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:47.251726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:47.251735 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:45:47.251789 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:47.251790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:47.251814 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:47.251871 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:45:47.251990 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.252535 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.253241 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.253444 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.253498 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.253624 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:45:47.253716 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:45:47.253781 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:47.253857 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 04:45:47.253874 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 04:45:47.255720 15808 raft_consensus.cc:1261] T 49e7f32b8db444088bf73aabd8cce382 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 1 FOLLOWER]: Refusing update from remote peer bdbad0e08ce14c83ab0de7783bdddd09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:45:47.255774 16076 raft_consensus.cc:1261] T 49e7f32b8db444088bf73aabd8cce382 P e6d19702051c4ea8bc3c0319416ef50e [term 1 FOLLOWER]: Refusing update from remote peer bdbad0e08ce14c83ab0de7783bdddd09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:45:47.256083 16149 consensus_queue.cc:1035] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:45:47.256250 16144 consensus_queue.cc:1035] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:45:47.259006 16154 mvcc.cc:205] Tried to move back new op lower bound from 7334177780756897792 to 7334177780710158336. Current Snapshot: MvccSnapshot[applied={T|T < 7334177780756897792}]
I20260928 04:45:47.261574 15663 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:45:47.261685 15663 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:45:47.262530 15663 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c: Sending DeleteTablet for 3 replicas of tablet 49e7f32b8db444088bf73aabd8cce382
I20260928 04:45:47.262893 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 15725
I20260928 04:45:47.262923 16056 tablet_service.cc:1514] Processing DeleteTablet for tablet 49e7f32b8db444088bf73aabd8cce382 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:45:47 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:57228
I20260928 04:45:47.263005 15788 tablet_service.cc:1514] Processing DeleteTablet for tablet 49e7f32b8db444088bf73aabd8cce382 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:45:47 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:49400
I20260928 04:45:47.263418 15922 tablet_service.cc:1514] Processing DeleteTablet for tablet 49e7f32b8db444088bf73aabd8cce382 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:45:47 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:44348
I20260928 04:45:47.263644 16163 tablet_replica.cc:330] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09: stopping tablet replica
I20260928 04:45:47.263743 16163 raft_consensus.cc:2229] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:45:47.263833 16163 raft_consensus.cc:2258] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:45:47.264226 16162 tablet_replica.cc:330] T 49e7f32b8db444088bf73aabd8cce382 P e6d19702051c4ea8bc3c0319416ef50e: stopping tablet replica
I20260928 04:45:47.264292 16163 ts_tablet_manager.cc:1905] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:45:47.264330 16162 raft_consensus.cc:2229] T 49e7f32b8db444088bf73aabd8cce382 P e6d19702051c4ea8bc3c0319416ef50e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:45:47.264424 16162 raft_consensus.cc:2258] T 49e7f32b8db444088bf73aabd8cce382 P e6d19702051c4ea8bc3c0319416ef50e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:45:47.264853 16162 ts_tablet_manager.cc:1905] T 49e7f32b8db444088bf73aabd8cce382 P e6d19702051c4ea8bc3c0319416ef50e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:45:47.266291 16163 ts_tablet_manager.cc:1918] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:45:47.266359 16163 log.cc:1196] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/wal/wals/49e7f32b8db444088bf73aabd8cce382
I20260928 04:45:47.266577 16162 ts_tablet_manager.cc:1918] T 49e7f32b8db444088bf73aabd8cce382 P e6d19702051c4ea8bc3c0319416ef50e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:45:47.266629 16162 log.cc:1196] T 49e7f32b8db444088bf73aabd8cce382 P e6d19702051c4ea8bc3c0319416ef50e: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/wal/wals/49e7f32b8db444088bf73aabd8cce382
I20260928 04:45:47.266669 16163 ts_tablet_manager.cc:1939] T 49e7f32b8db444088bf73aabd8cce382 P bdbad0e08ce14c83ab0de7783bdddd09: Deleting consensus metadata
I20260928 04:45:47.266848 16162 ts_tablet_manager.cc:1939] T 49e7f32b8db444088bf73aabd8cce382 P e6d19702051c4ea8bc3c0319416ef50e: Deleting consensus metadata
I20260928 04:45:47.267063 15647 catalog_manager.cc:4841] TS bdbad0e08ce14c83ab0de7783bdddd09 (127.1.226.194:32989): tablet 49e7f32b8db444088bf73aabd8cce382 (table test-table [id=70361d1b546647c2b4089ce2fdd7e585]) successfully deleted
I20260928 04:45:47.267339 15647 catalog_manager.cc:4841] TS e6d19702051c4ea8bc3c0319416ef50e (127.1.226.195:39013): tablet 49e7f32b8db444088bf73aabd8cce382 (table test-table [id=70361d1b546647c2b4089ce2fdd7e585]) successfully deleted
W20260928 04:45:47.268535 15647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.226.193:43663 (error 108)
I20260928 04:45:47.268826 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 15859
I20260928 04:45:47.269491 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.268989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:43663 (local address 127.0.0.1:49406)
0928 04:45:47.269415 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:47.269457 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.193:43663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404}
W20260928 04:45:47.269634 15647 catalog_manager.cc:4584] TS 3cfb71b57e6648cbb0786b6250f3e5c9 (127.1.226.193:43663): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 49e7f32b8db444088bf73aabd8cce382: Network error: Client connection negotiation failed: client connection to 127.1.226.193:43663: connect: Connection refused (error 111)
I20260928 04:45:47.273901 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 15993
I20260928 04:45:47.278393 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 15631
I20260928 04:45:47.283998 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:36905
--webserver_interface=127.1.226.254
--webserver_port=40761
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:36905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:45:47.368860 16165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:45:47.369113 16165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:45:47.369161 16165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:45:47.370363 16165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:45:47.370446 16165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:45:47.370473 16165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:45:47.370507 16165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:45:47.370540 16165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:45:47.372040 16165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:36905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:36905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=40761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:45:47.372352 16165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:45:47.372601 16165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:45:47.381669 16171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:47.381785 16165 server_base.cc:1034] running on GCE node
W20260928 04:45:47.381671 16174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:47.381665 16172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:47.382289 16165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:45:47.382602 16165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:45:47.383764 16165 hybrid_clock.cc:648] HybridClock initialized: now 1790570747383741 us; error 38 us; skew 500 ppm
Sep 28 04:45:47 dist-test-slave-j5c3 krb5kdc[15618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570747, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:45:47.386596 16165 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:45:47.387144 16165 webserver.cc:460] Webserver started at http://127.1.226.254:40761/ using document root <none> and password file <none>
I20260928 04:45:47.387408 16165 fs_manager.cc:359] Metadata directory not provided
I20260928 04:45:47.387490 16165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:45:47.389106 16165 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:45:47.389842 16181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:47.390058 16165 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:45:47.390167 16165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "1a0ae1415dd94dc483884d84a07bcb1c"
format_stamp: "Formatted at 2026-09-28 04:45:46 on dist-test-slave-j5c3"
server_key: "dec1fd3266d6b46b150360f071114e19"
server_key_iv: "fda9c95cbeddf048fe88bab4893da9d9"
server_key_version: "encryptionkey@0"
I20260928 04:45:47.390602 16165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:45:47.401139 16165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:45:47.403548 16165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:45:47.403705 16165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:45:47.407907 16165 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:36905
I20260928 04:45:47.407967 16233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:36905 every 8 connection(s)
I20260928 04:45:47.408396 16165 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:45:47.409122 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 16165
I20260928 04:45:47.409437 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:43663
--local_ip_for_outbound_sockets=127.1.226.193
--tserver_master_addrs=127.1.226.254:36905
--webserver_port=38635
--webserver_interface=127.1.226.193
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:45:47.410182 16234 sys_catalog.cc:263] Verifying existing consensus state
I20260928 04:45:47.411151 16234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c: Bootstrap starting.
I20260928 04:45:47.419605 16234 log.cc:824] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c: Log is configured to *not* fsync() on all Append() calls
I20260928 04:45:47.421660 16234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:45:47.422030 16234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c: Bootstrap complete.
I20260928 04:45:47.424437 16234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0ae1415dd94dc483884d84a07bcb1c" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 36905 } }
I20260928 04:45:47.424669 16234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a0ae1415dd94dc483884d84a07bcb1c, State: Initialized, Role: FOLLOWER
I20260928 04:45:47.424837 16234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0ae1415dd94dc483884d84a07bcb1c" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 36905 } }
I20260928 04:45:47.424904 16234 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:45:47.424958 16234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:45:47.425006 16234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [term 1 FOLLOWER]: Advancing to term 2
I20260928 04:45:47.426075 16234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0ae1415dd94dc483884d84a07bcb1c" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 36905 } }
I20260928 04:45:47.426162 16234 leader_election.cc:302] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a0ae1415dd94dc483884d84a07bcb1c; no voters:
I20260928 04:45:47.426385 16234 leader_election.cc:288] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 04:45:47.426421 16238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [term 2 FOLLOWER]: Leader election won for term 2
I20260928 04:45:47.426530 16238 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [term 2 LEADER]: Becoming Leader. State: Replica: 1a0ae1415dd94dc483884d84a07bcb1c, State: Running, Role: LEADER
I20260928 04:45:47.426669 16234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:45:47.426689 16238 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0ae1415dd94dc483884d84a07bcb1c" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 36905 } }
I20260928 04:45:47.427107 16239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1a0ae1415dd94dc483884d84a07bcb1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0ae1415dd94dc483884d84a07bcb1c" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 36905 } } }
I20260928 04:45:47.427444 16239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [sys.catalog]: This master's current role is: LEADER
I20260928 04:45:47.427379 16240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a0ae1415dd94dc483884d84a07bcb1c. Latest consensus state: current_term: 2 leader_uuid: "1a0ae1415dd94dc483884d84a07bcb1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0ae1415dd94dc483884d84a07bcb1c" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 36905 } } }
I20260928 04:45:47.427664 16248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:45:47.427682 16240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [sys.catalog]: This master's current role is: LEADER
I20260928 04:45:47.428580 16248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:45:47.428742 16248 catalog_manager.cc:1259] Loaded cluster ID: c8e908171bd24a1ca57f81a48a3d9fb5
I20260928 04:45:47.428984 16248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:45:47.429234 16248 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 04:45:47.433902 16248 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 04:45:47.436578 16248 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:45:47.436803 16248 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c: Loaded TSK: 0
I20260928 04:45:47.437044 16248 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 04:45:47.494993 16236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:45:47.495236 16236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:45:47.495285 16236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:45:47.496565 16236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:45:47.496626 16236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:45:47.496672 16236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:45:47.498162 16236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:43663
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=38635
--tserver_master_addrs=127.1.226.254:36905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:45:47.498443 16236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:45:47.498677 16236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:45:47.504758 16264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:47.504756 16262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:47.504769 16261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:47.505314 16236 server_base.cc:1034] running on GCE node
I20260928 04:45:47.505548 16236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:45:47.505838 16236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:45:47.507015 16236 hybrid_clock.cc:648] HybridClock initialized: now 1790570747506997 us; error 35 us; skew 500 ppm
Sep 28 04:45:47 dist-test-slave-j5c3 krb5kdc[15618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570747, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:45:47.510071 16236 init.cc:377] Logged in from keytab as kudu/127.1.226.193@KRBTEST.COM (short username kudu)
I20260928 04:45:47.510610 16236 webserver.cc:460] Webserver started at http://127.1.226.193:38635/ using document root <none> and password file <none>
I20260928 04:45:47.510854 16236 fs_manager.cc:359] Metadata directory not provided
I20260928 04:45:47.510938 16236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:45:47.512446 16236 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:45:47.513231 16271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:47.513418 16236 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:45:47.513531 16236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "3cfb71b57e6648cbb0786b6250f3e5c9"
format_stamp: "Formatted at 2026-09-28 04:45:46 on dist-test-slave-j5c3"
server_key: "0833c0510044e057eb27d0b290dd40e8"
server_key_iv: "263e9b97f9e55a381c914d87c669cb6b"
server_key_version: "encryptionkey@0"
I20260928 04:45:47.513909 16236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:45:47.525323 16236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:45:47.527819 16236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:45:47.527987 16236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:45:47.528286 16236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:45:47.528771 16278 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 04:45:47.530880 16236 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 04:45:47.530964 16236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 04:45:47.531013 16236 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 04:45:47.532016 16236 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 04:45:47.532086 16236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260928 04:45:47.532337 16278 tablet_bootstrap.cc:492] T 49e7f32b8db444088bf73aabd8cce382 P 3cfb71b57e6648cbb0786b6250f3e5c9: Bootstrap starting.
I20260928 04:45:47.538856 16236 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:43663
I20260928 04:45:47.539237 16236 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
I20260928 04:45:47.543694 16278 log.cc:824] T 49e7f32b8db444088bf73aabd8cce382 P 3cfb71b57e6648cbb0786b6250f3e5c9: Log is configured to *not* fsync() on all Append() calls
I20260928 04:45:47.544970 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 16236
I20260928 04:45:47.545265 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:32989
--local_ip_for_outbound_sockets=127.1.226.194
--tserver_master_addrs=127.1.226.254:36905
--webserver_port=42245
--webserver_interface=127.1.226.194
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:45:47.547415 16385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:43663 every 8 connection(s)
I20260928 04:45:47.547423 16278 tablet_bootstrap.cc:492] T 49e7f32b8db444088bf73aabd8cce382 P 3cfb71b57e6648cbb0786b6250f3e5c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:45:47.547883 16278 tablet_bootstrap.cc:492] T 49e7f32b8db444088bf73aabd8cce382 P 3cfb71b57e6648cbb0786b6250f3e5c9: Bootstrap complete.
I20260928 04:45:47.549129 16278 ts_tablet_manager.cc:1397] T 49e7f32b8db444088bf73aabd8cce382 P 3cfb71b57e6648cbb0786b6250f3e5c9: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.014s
I20260928 04:45:47.550714 16278 raft_consensus.cc:348] T 49e7f32b8db444088bf73aabd8cce382 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 } } peers { permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 } } peers { permanent_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32989 } }
I20260928 04:45:47.551081 16278 raft_consensus.cc:729] T 49e7f32b8db444088bf73aabd8cce382 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cfb71b57e6648cbb0786b6250f3e5c9, State: Initialized, Role: FOLLOWER
I20260928 04:45:47.551249 16278 consensus_queue.cc:260] T 49e7f32b8db444088bf73aabd8cce382 P 3cfb71b57e6648cbb0786b6250f3e5c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 } } peers { permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 } } peers { permanent_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32989 } }
I20260928 04:45:47.551609 16278 ts_tablet_manager.cc:1428] T 49e7f32b8db444088bf73aabd8cce382 P 3cfb71b57e6648cbb0786b6250f3e5c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
Sep 28 04:45:47 dist-test-slave-j5c3 krb5kdc[15618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570747, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:45:47.557315 16388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.540009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.193:41793)
0928 04:45:47.540216 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:47.540269 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:47.540732 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:47.544682 (+ 3950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:47.544687 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:47.545110 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:45:47.545431 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:47.545439 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.547154 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.547156 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:47.548645 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:47.548649 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.549503 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.549505 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:47.549656 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.550276 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:45:47.550294 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:45:47.552358 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:45:47.555057 (+ 2699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:47.555067 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:47.555070 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:47.555425 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:47.555813 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:47.555814 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:47.555814 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:47.555974 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:47.556581 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:45:47.556588 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:45:47.556768 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 04:45:47.556893 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 04:45:47.557084 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":107,"threads_started":1}
I20260928 04:45:47.557466 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.540064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:41793 (local address 127.1.226.254:36905)
0928 04:45:47.544066 (+ 4002us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:47.544078 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:47.544125 (+ 47us) server_negotiation.cc:407] Connection header received
0928 04:45:47.544299 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:47.544307 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:47.544371 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:47.544530 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:47.545590 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.547006 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.548763 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.549368 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.549503 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.552495 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:47.552500 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:47.552500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:47.552515 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:47.554938 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:47.555554 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:47.555558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:47.555559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:47.555713 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:47.556065 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:47.556066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:47.556066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:47.556493 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:47.556597 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:47.557009 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 04:45:47.557186 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3933,"thread_start_us":106,"threads_started":1}
I20260928 04:45:47.559407 16386 heartbeater.cc:347] Connected to a master server at 127.1.226.254:36905
I20260928 04:45:47.559533 16386 heartbeater.cc:464] Registering TS with master...
I20260928 04:45:47.559751 16386 heartbeater.cc:511] Master 127.1.226.254:36905 requested a full tablet report, sending...
I20260928 04:45:47.560665 16198 ts_manager.cc:194] Registered new tserver with Master: 3cfb71b57e6648cbb0786b6250f3e5c9 (127.1.226.193:43663)
I20260928 04:45:47.561950 16198 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.193@KRBTEST.COM'} at 127.1.226.193:41793
I20260928 04:45:47.563109 16386 heartbeater.cc:503] Master 127.1.226.254:36905 was elected leader, sending a full tablet report...
Sep 28 04:45:47 dist-test-slave-j5c3 krb5kdc[15618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570747, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for kudu/127.1.226.193@KRBTEST.COM
I20260928 04:45:47.571949 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.561419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:43663 (local address 127.0.0.1:49412)
0928 04:45:47.561925 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:47.561967 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:47.562039 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:47.562378 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:47.562383 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:47.562740 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:45:47.562802 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:47.562805 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.564110 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.564112 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:47.565107 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:47.565111 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.565783 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.565784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:47.565875 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.566492 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:45:47.566498 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:45:47.568349 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:45:47.570588 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:47.570598 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:47.570600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:47.570909 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:47.571164 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:47.571165 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:47.571165 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:47.571253 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:47.571616 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:45:47.571622 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:45:47.571672 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 04:45:47.571742 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 04:45:47.571872 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":118,"threads_started":1}
I20260928 04:45:47.572008 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.561729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49412 (local address 127.1.226.193:43663)
0928 04:45:47.562171 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:47.562176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:47.562186 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:45:47.562227 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:47.562229 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:47.562242 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:47.562302 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:47.562913 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.564015 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.565206 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.565686 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.565768 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.568458 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:47.568474 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:47.568475 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:47.568484 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:47.570476 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:47.571007 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:47.571011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:47.571011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:47.571084 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:47.571325 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:47.571326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:47.571326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:47.571545 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:47.571631 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:47.571811 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 04:45:47.571918 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":82,"threads_started":1}
I20260928 04:45:47.572512 16320 tablet_service.cc:1514] Processing DeleteTablet for tablet 49e7f32b8db444088bf73aabd8cce382 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:45:47 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:49412
I20260928 04:45:47.573179 16395 tablet_replica.cc:330] stopping tablet replica
I20260928 04:45:47.573310 16395 raft_consensus.cc:2229] T 49e7f32b8db444088bf73aabd8cce382 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:45:47.573396 16395 raft_consensus.cc:2258] T 49e7f32b8db444088bf73aabd8cce382 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:45:47.573691 16395 ts_tablet_manager.cc:1905] T 49e7f32b8db444088bf73aabd8cce382 P 3cfb71b57e6648cbb0786b6250f3e5c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:45:47.575668 16395 ts_tablet_manager.cc:1918] T 49e7f32b8db444088bf73aabd8cce382 P 3cfb71b57e6648cbb0786b6250f3e5c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:45:47.575763 16395 log.cc:1196] T 49e7f32b8db444088bf73aabd8cce382 P 3cfb71b57e6648cbb0786b6250f3e5c9: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/wal/wals/49e7f32b8db444088bf73aabd8cce382
I20260928 04:45:47.575991 16395 ts_tablet_manager.cc:1939] T 49e7f32b8db444088bf73aabd8cce382 P 3cfb71b57e6648cbb0786b6250f3e5c9: Deleting consensus metadata
I20260928 04:45:47.576582 16182 catalog_manager.cc:4841] TS 3cfb71b57e6648cbb0786b6250f3e5c9 (127.1.226.193:43663): tablet 49e7f32b8db444088bf73aabd8cce382 (table test-table [id=70361d1b546647c2b4089ce2fdd7e585]) successfully deleted
W20260928 04:45:47.638624 16391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:45:47.638842 16391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:45:47.638866 16391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:45:47.640054 16391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:45:47.640096 16391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:45:47.640122 16391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:45:47.641628 16391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:32989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=42245
--tserver_master_addrs=127.1.226.254:36905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:45:47.641886 16391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:45:47.642133 16391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:45:47.647303 16401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:47.647315 16403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:47.647385 16400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:47.647495 16391 server_base.cc:1034] running on GCE node
I20260928 04:45:47.647847 16391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:45:47.648124 16391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:45:47.649330 16391 hybrid_clock.cc:648] HybridClock initialized: now 1790570747649310 us; error 39 us; skew 500 ppm
Sep 28 04:45:47 dist-test-slave-j5c3 krb5kdc[15618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570747, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:45:47.651798 16391 init.cc:377] Logged in from keytab as kudu/127.1.226.194@KRBTEST.COM (short username kudu)
I20260928 04:45:47.652406 16391 webserver.cc:460] Webserver started at http://127.1.226.194:42245/ using document root <none> and password file <none>
I20260928 04:45:47.652663 16391 fs_manager.cc:359] Metadata directory not provided
I20260928 04:45:47.652747 16391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:45:47.654158 16391 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:45:47.654968 16410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:47.655158 16391 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:45:47.655284 16391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "bdbad0e08ce14c83ab0de7783bdddd09"
format_stamp: "Formatted at 2026-09-28 04:45:46 on dist-test-slave-j5c3"
server_key: "9c539262f6d69ee1539b64df0cb6e0fc"
server_key_iv: "58630404ba18b4b6605f8146f62c191e"
server_key_version: "encryptionkey@0"
I20260928 04:45:47.655656 16391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:45:47.666551 16391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:45:47.668622 16391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:45:47.668798 16391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:45:47.669042 16391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:45:47.669370 16391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:45:47.669447 16391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:47.669492 16391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:45:47.669521 16391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:47.675892 16391 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:32989
I20260928 04:45:47.675936 16523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:32989 every 8 connection(s)
I20260928 04:45:47.676407 16391 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
Sep 28 04:45:47 dist-test-slave-j5c3 krb5kdc[15618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570747, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:45:47.684281 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 16391
I20260928 04:45:47.684612 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:39013
--local_ip_for_outbound_sockets=127.1.226.195
--tserver_master_addrs=127.1.226.254:36905
--webserver_port=44139
--webserver_interface=127.1.226.195
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:45:47.685968 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.677224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:44253 (local address 127.1.226.254:36905)
0928 04:45:47.677312 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:47.677315 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:47.677523 (+ 208us) server_negotiation.cc:407] Connection header received
0928 04:45:47.678027 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:47.678028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:47.678042 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:47.678103 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:47.679008 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.679510 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.680543 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.680762 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.680822 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.682890 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:47.682892 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:47.682892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:47.682898 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:47.684525 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:47.684977 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:47.684979 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:47.684979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:47.685042 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:47.685277 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:47.685278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:47.685279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:47.685424 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:47.685492 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:47.685737 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 04:45:47.685837 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 04:45:47.686108 16526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.677205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.194:44253)
0928 04:45:47.677396 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:47.677434 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:47.677914 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:47.678153 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:47.678156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:47.678723 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:45:47.678917 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:47.678922 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.679597 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.679598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:47.680463 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:47.680466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.680838 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.680839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:47.680935 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.681412 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:45:47.681420 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:45:47.682794 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:45:47.684600 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:47.684606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:47.684608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:47.684904 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:47.685117 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:47.685118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:47.685118 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:47.685215 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:47.685488 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:45:47.685492 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:45:47.685607 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 04:45:47.685688 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 04:45:47.685829 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":112,"threads_started":1}
I20260928 04:45:47.686723 16524 heartbeater.cc:347] Connected to a master server at 127.1.226.254:36905
I20260928 04:45:47.686836 16524 heartbeater.cc:464] Registering TS with master...
I20260928 04:45:47.687017 16524 heartbeater.cc:511] Master 127.1.226.254:36905 requested a full tablet report, sending...
I20260928 04:45:47.687479 16198 ts_manager.cc:194] Registered new tserver with Master: bdbad0e08ce14c83ab0de7783bdddd09 (127.1.226.194:32989)
I20260928 04:45:47.687902 16198 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.194@KRBTEST.COM'} at 127.1.226.194:44253
W20260928 04:45:47.767962 16527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:45:47.768250 16527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:45:47.768291 16527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:45:47.769510 16527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:45:47.769594 16527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:45:47.769642 16527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:45:47.771157 16527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:39013
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=44139
--tserver_master_addrs=127.1.226.254:36905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:45:47.771448 16527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:45:47.771776 16527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:45:47.776957 16532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:47.776973 16533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:47.777112 16527 server_base.cc:1034] running on GCE node
W20260928 04:45:47.776970 16535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:47.777438 16527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:45:47.777801 16527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:45:47.778965 16527 hybrid_clock.cc:648] HybridClock initialized: now 1790570747778954 us; error 30 us; skew 500 ppm
Sep 28 04:45:47 dist-test-slave-j5c3 krb5kdc[15618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570747, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:45:47.781737 16527 init.cc:377] Logged in from keytab as kudu/127.1.226.195@KRBTEST.COM (short username kudu)
I20260928 04:45:47.782207 16527 webserver.cc:460] Webserver started at http://127.1.226.195:44139/ using document root <none> and password file <none>
I20260928 04:45:47.782436 16527 fs_manager.cc:359] Metadata directory not provided
I20260928 04:45:47.782492 16527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:45:47.783922 16527 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:45:47.784684 16542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:47.784940 16527 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:45:47.785061 16527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "e6d19702051c4ea8bc3c0319416ef50e"
format_stamp: "Formatted at 2026-09-28 04:45:47 on dist-test-slave-j5c3"
server_key: "48bb458b125d07faccd8ca28feb9d2e2"
server_key_iv: "0c219606240e6d95f82d8667ade36b6e"
server_key_version: "encryptionkey@0"
I20260928 04:45:47.785432 16527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:45:47.797138 16527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:45:47.799283 16527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:45:47.799503 16527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:45:47.799769 16527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:45:47.800091 16527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:45:47.800145 16527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:47.800235 16527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:45:47.800259 16527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:47.806334 16527 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:39013
I20260928 04:45:47.806381 16655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:39013 every 8 connection(s)
I20260928 04:45:47.806850 16527 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
I20260928 04:45:47.810477 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 16527
Sep 28 04:45:47 dist-test-slave-j5c3 krb5kdc[15618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570747, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:45:47.817698 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.807713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:41639 (local address 127.1.226.254:36905)
0928 04:45:47.807790 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:47.807792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:47.808002 (+ 210us) server_negotiation.cc:407] Connection header received
0928 04:45:47.808512 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:47.808513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:47.808524 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:47.808587 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:47.809546 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.810037 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.811412 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.811757 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.811836 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.814350 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:47.814352 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:47.814352 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:47.814359 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:47.816013 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:47.816482 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:47.816483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:47.816484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:47.816545 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:47.816806 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:47.816807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:47.816808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:47.817050 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:47.817163 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:47.817503 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 04:45:47.817595 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 04:45:47.817943 16658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.807703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.195:41639)
0928 04:45:47.807858 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:47.807902 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:47.808421 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:47.808642 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:47.808645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:47.809233 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:45:47.809458 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:47.809463 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.810127 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.810128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:47.811333 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:47.811336 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.811860 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.811861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:47.812005 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.812537 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:45:47.812547 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:45:47.814235 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:45:47.816099 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:47.816106 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:47.816109 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:47.816388 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:47.816612 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:47.816613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:47.816614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:47.816731 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:47.817154 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:45:47.817162 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:45:47.817342 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 04:45:47.817463 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 04:45:47.817647 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
I20260928 04:45:47.818231 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.810925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36398 (local address 127.1.226.254:36905)
0928 04:45:47.811060 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:47.811062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:47.811073 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:45:47.811103 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:47.811104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:47.811115 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:47.811186 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:47.811866 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.812555 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.813330 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.813517 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.813571 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.814784 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:47.814785 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:47.814785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:47.814793 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:47.816577 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:47.816989 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:47.816990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:47.816990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:47.817054 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:47.817330 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:47.817331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:47.817331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:47.817457 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:47.817646 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:47.817692 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 04:45:47.818126 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20260928 04:45:47.818531 16656 heartbeater.cc:347] Connected to a master server at 127.1.226.254:36905
I20260928 04:45:47.818647 16656 heartbeater.cc:464] Registering TS with master...
I20260928 04:45:47.818849 16656 heartbeater.cc:511] Master 127.1.226.254:36905 requested a full tablet report, sending...
I20260928 04:45:47.819226 16198 ts_manager.cc:194] Registered new tserver with Master: e6d19702051c4ea8bc3c0319416ef50e (127.1.226.195:39013)
I20260928 04:45:47.819590 16198 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.195@KRBTEST.COM'} at 127.1.226.195:41639
I20260928 04:45:47.830087 1931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:45:47.833387 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.830522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36408 (local address 127.1.226.254:36905)
0928 04:45:47.830602 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:47.830604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:47.830678 (+ 74us) server_negotiation.cc:407] Connection header received
0928 04:45:47.830784 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:47.830785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:47.830823 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:47.830892 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:45:47.831010 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.831512 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.832470 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.832672 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.832780 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.832957 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:45:47.833070 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:45:47.833150 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:47.833238 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 04:45:47.833264 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 04:45:47.833813 16198 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:36408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:45:47.835476 16198 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:45:47.838524 16320 tablet_service.cc:1467] Processing CreateTablet for tablet 1caf2c2be98d4df5b802e2789cddca73 (DEFAULT_TABLE table=test-table [id=bb95af94d7494381a353127bcb0f0416]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:45:47.838856 16320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1caf2c2be98d4df5b802e2789cddca73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:45:47.840656 16278 tablet_bootstrap.cc:492] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9: Bootstrap starting.
I20260928 04:45:47.841007 16278 tablet_bootstrap.cc:654] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9: Neither blocks nor log segments found. Creating new log.
I20260928 04:45:47.841614 16278 tablet_bootstrap.cc:492] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9: No bootstrap required, opened a new log
I20260928 04:45:47.841686 16278 ts_tablet_manager.cc:1397] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 04:45:47.841998 16278 raft_consensus.cc:348] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32989 } } peers { permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 } } peers { permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 } }
I20260928 04:45:47.842082 16278 raft_consensus.cc:374] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:45:47.842139 16278 raft_consensus.cc:729] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cfb71b57e6648cbb0786b6250f3e5c9, State: Initialized, Role: FOLLOWER
I20260928 04:45:47.842202 16278 consensus_queue.cc:260] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32989 } } peers { permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 } } peers { permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 } }
I20260928 04:45:47.842303 16278 ts_tablet_manager.cc:1428] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 04:45:47.842541 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.837973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:32989 (local address 127.0.0.1:44368)
0928 04:45:47.838038 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:47.838055 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:47.838112 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:47.838573 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:47.838574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:47.838576 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:45:47.838634 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:47.838636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.839944 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.839944 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:47.841305 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:47.841308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.842212 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.842213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:47.842315 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.842346 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 04:45:47.842384 (+ 38us) client_negotiation.cc:240] Negotiation successful
0928 04:45:47.842445 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":5}
I20260928 04:45:47.843040 16663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.838133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44368 (local address 127.1.226.194:32989)
0928 04:45:47.838370 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:47.838376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:47.838388 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:45:47.838432 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:47.838434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:47.838458 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:47.838566 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:45:47.838752 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.839843 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.841411 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.842082 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.842747 (+ 665us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.842831 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:47.842863 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 04:45:47.842914 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":110,"threads_started":1}
I20260928 04:45:47.843539 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.838063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:39013 (local address 127.0.0.1:57230)
0928 04:45:47.838228 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:47.838238 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:47.838285 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:47.838673 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:47.838674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:47.838676 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:45:47.838716 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:47.838717 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.839852 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.839854 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:47.841379 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:47.841381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.843278 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.843279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:47.843347 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.843351 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:45:47.843403 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 04:45:47.843446 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":23,"thread_start_us":124,"threads_started":1}
I20260928 04:45:47.843832 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.838222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57230 (local address 127.1.226.195:39013)
0928 04:45:47.838447 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:47.838452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:47.838462 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:45:47.838505 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:47.838507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:47.838521 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:47.838599 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:45:47.838800 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.839730 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.841466 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.843068 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.843182 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.843241 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:47.843694 (+ 453us) server_negotiation.cc:299] Negotiation successful
0928 04:45:47.843742 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":87,"threads_started":1}
I20260928 04:45:47.843819 16458 tablet_service.cc:1467] Processing CreateTablet for tablet 1caf2c2be98d4df5b802e2789cddca73 (DEFAULT_TABLE table=test-table [id=bb95af94d7494381a353127bcb0f0416]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:45:47.844259 16458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1caf2c2be98d4df5b802e2789cddca73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:45:47.844584 16590 tablet_service.cc:1467] Processing CreateTablet for tablet 1caf2c2be98d4df5b802e2789cddca73 (DEFAULT_TABLE table=test-table [id=bb95af94d7494381a353127bcb0f0416]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:45:47.844887 16590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1caf2c2be98d4df5b802e2789cddca73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:45:47.847549 16667 tablet_bootstrap.cc:492] T 1caf2c2be98d4df5b802e2789cddca73 P e6d19702051c4ea8bc3c0319416ef50e: Bootstrap starting.
I20260928 04:45:47.847507 16666 tablet_bootstrap.cc:492] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09: Bootstrap starting.
I20260928 04:45:47.848225 16667 tablet_bootstrap.cc:654] T 1caf2c2be98d4df5b802e2789cddca73 P e6d19702051c4ea8bc3c0319416ef50e: Neither blocks nor log segments found. Creating new log.
I20260928 04:45:47.848275 16666 tablet_bootstrap.cc:654] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09: Neither blocks nor log segments found. Creating new log.
I20260928 04:45:47.848724 16667 log.cc:824] T 1caf2c2be98d4df5b802e2789cddca73 P e6d19702051c4ea8bc3c0319416ef50e: Log is configured to *not* fsync() on all Append() calls
I20260928 04:45:47.848785 16666 log.cc:824] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09: Log is configured to *not* fsync() on all Append() calls
I20260928 04:45:47.849529 16667 tablet_bootstrap.cc:492] T 1caf2c2be98d4df5b802e2789cddca73 P e6d19702051c4ea8bc3c0319416ef50e: No bootstrap required, opened a new log
I20260928 04:45:47.849550 16666 tablet_bootstrap.cc:492] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09: No bootstrap required, opened a new log
I20260928 04:45:47.849612 16667 ts_tablet_manager.cc:1397] T 1caf2c2be98d4df5b802e2789cddca73 P e6d19702051c4ea8bc3c0319416ef50e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:45:47.849632 16666 ts_tablet_manager.cc:1397] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 04:45:47.851161 16667 raft_consensus.cc:348] T 1caf2c2be98d4df5b802e2789cddca73 P e6d19702051c4ea8bc3c0319416ef50e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32989 } } peers { permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 } } peers { permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 } }
I20260928 04:45:47.851322 16667 raft_consensus.cc:374] T 1caf2c2be98d4df5b802e2789cddca73 P e6d19702051c4ea8bc3c0319416ef50e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:45:47.851367 16667 raft_consensus.cc:729] T 1caf2c2be98d4df5b802e2789cddca73 P e6d19702051c4ea8bc3c0319416ef50e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6d19702051c4ea8bc3c0319416ef50e, State: Initialized, Role: FOLLOWER
I20260928 04:45:47.851269 16666 raft_consensus.cc:348] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32989 } } peers { permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 } } peers { permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 } }
I20260928 04:45:47.851470 16666 raft_consensus.cc:374] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:45:47.851490 16667 consensus_queue.cc:260] T 1caf2c2be98d4df5b802e2789cddca73 P e6d19702051c4ea8bc3c0319416ef50e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32989 } } peers { permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 } } peers { permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 } }
I20260928 04:45:47.851543 16666 raft_consensus.cc:729] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdbad0e08ce14c83ab0de7783bdddd09, State: Initialized, Role: FOLLOWER
I20260928 04:45:47.851670 16666 consensus_queue.cc:260] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32989 } } peers { permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 } } peers { permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 } }
I20260928 04:45:47.851805 16667 ts_tablet_manager.cc:1428] T 1caf2c2be98d4df5b802e2789cddca73 P e6d19702051c4ea8bc3c0319416ef50e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:45:47.851809 16656 heartbeater.cc:503] Master 127.1.226.254:36905 was elected leader, sending a full tablet report...
I20260928 04:45:47.851984 16666 ts_tablet_manager.cc:1428] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 04:45:47.852061 16524 heartbeater.cc:503] Master 127.1.226.254:36905 was elected leader, sending a full tablet report...
I20260928 04:45:47.925268 16671 raft_consensus.cc:482] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:45:47.925403 16671 raft_consensus.cc:504] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32989 } } peers { permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 } } peers { permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 } }
I20260928 04:45:47.925830 16671 leader_election.cc:288] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6d19702051c4ea8bc3c0319416ef50e (127.1.226.195:39013), 3cfb71b57e6648cbb0786b6250f3e5c9 (127.1.226.193:43663)
W20260928 04:45:47.927011 16525 tablet.cc:2369] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:45:47.929464 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.926301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:39365 (local address 127.1.226.193:43663)
0928 04:45:47.926373 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:47.926376 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:47.926512 (+ 136us) server_negotiation.cc:407] Connection header received
0928 04:45:47.926585 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:47.926586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:47.926633 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:47.926697 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:45:47.926934 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.927490 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.928675 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.929111 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.929202 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.929317 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:47.929337 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 04:45:47.929376 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 04:45:47.929492 16672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.926236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:43663 (local address 127.1.226.194:39365)
0928 04:45:47.926439 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:47.926448 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:47.926527 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:47.926701 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:47.926701 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:47.926703 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:45:47.926808 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:47.926809 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.927575 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.927576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:47.928583 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:47.928585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.929200 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.929201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:47.929268 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.929272 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:45:47.929321 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 04:45:47.929365 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":141,"threads_started":1}
I20260928 04:45:47.929504 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.926133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:55901 (local address 127.1.226.195:39013)
0928 04:45:47.926195 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:47.926198 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:47.926228 (+ 30us) server_negotiation.cc:407] Connection header received
0928 04:45:47.926349 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:47.926350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:47.926360 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:47.926434 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:45:47.926650 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.927303 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.928604 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.929106 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.929205 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.929216 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:47.929391 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 04:45:47.929433 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 04:45:47.929546 16526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.926066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:39013 (local address 127.1.226.194:55901)
0928 04:45:47.926142 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:47.926171 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:47.926248 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:47.926438 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:47.926440 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:47.926442 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:45:47.926522 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:47.926524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.927397 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.927398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:47.928511 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:47.928513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.929206 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.929207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:47.929298 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.929301 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:45:47.929374 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 04:45:47.929428 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":68}
I20260928 04:45:47.930042 16340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1caf2c2be98d4df5b802e2789cddca73" candidate_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" is_pre_election: true
I20260928 04:45:47.930066 16610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1caf2c2be98d4df5b802e2789cddca73" candidate_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6d19702051c4ea8bc3c0319416ef50e" is_pre_election: true
I20260928 04:45:47.930207 16610 raft_consensus.cc:2454] T 1caf2c2be98d4df5b802e2789cddca73 P e6d19702051c4ea8bc3c0319416ef50e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdbad0e08ce14c83ab0de7783bdddd09 in term 0.
I20260928 04:45:47.930212 16340 raft_consensus.cc:2454] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdbad0e08ce14c83ab0de7783bdddd09 in term 0.
I20260928 04:45:47.930418 16411 leader_election.cc:302] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdbad0e08ce14c83ab0de7783bdddd09, e6d19702051c4ea8bc3c0319416ef50e; no voters:
I20260928 04:45:47.930577 16671 raft_consensus.cc:2781] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:45:47.930655 16671 raft_consensus.cc:482] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:45:47.930673 16671 raft_consensus.cc:3037] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:45:47.931492 16671 raft_consensus.cc:504] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32989 } } peers { permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 } } peers { permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 } }
I20260928 04:45:47.931623 16671 leader_election.cc:288] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09 [CANDIDATE]: Term 1 election: Requested vote from peers e6d19702051c4ea8bc3c0319416ef50e (127.1.226.195:39013), 3cfb71b57e6648cbb0786b6250f3e5c9 (127.1.226.193:43663)
I20260928 04:45:47.931799 16610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1caf2c2be98d4df5b802e2789cddca73" candidate_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6d19702051c4ea8bc3c0319416ef50e"
I20260928 04:45:47.931872 16610 raft_consensus.cc:3037] T 1caf2c2be98d4df5b802e2789cddca73 P e6d19702051c4ea8bc3c0319416ef50e [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:45:47.931862 16340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1caf2c2be98d4df5b802e2789cddca73" candidate_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9"
I20260928 04:45:47.931957 16340 raft_consensus.cc:3037] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:45:47.932660 16610 raft_consensus.cc:2454] T 1caf2c2be98d4df5b802e2789cddca73 P e6d19702051c4ea8bc3c0319416ef50e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdbad0e08ce14c83ab0de7783bdddd09 in term 1.
I20260928 04:45:47.932726 16340 raft_consensus.cc:2454] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdbad0e08ce14c83ab0de7783bdddd09 in term 1.
I20260928 04:45:47.932873 16411 leader_election.cc:302] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdbad0e08ce14c83ab0de7783bdddd09, e6d19702051c4ea8bc3c0319416ef50e; no voters:
I20260928 04:45:47.932978 16671 raft_consensus.cc:2781] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:45:47.933159 16671 raft_consensus.cc:686] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09 [term 1 LEADER]: Becoming Leader. State: Replica: bdbad0e08ce14c83ab0de7783bdddd09, State: Running, Role: LEADER
I20260928 04:45:47.933269 16671 consensus_queue.cc:237] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32989 } } peers { permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 } } peers { permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 } }
I20260928 04:45:47.934129 16197 catalog_manager.cc:5495] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09 reported cstate change: term changed from 0 to 1, leader changed from <none> to bdbad0e08ce14c83ab0de7783bdddd09 (127.1.226.194). New cstate: current_term: 1 leader_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 } health_report { overall_health: UNKNOWN } } }
I20260928 04:45:47.959339 16663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.956901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44372 (local address 127.1.226.194:32989)
0928 04:45:47.956967 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:47.956969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:47.956979 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:45:47.957055 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:47.957057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:47.957083 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:47.957164 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:45:47.957295 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.957785 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.958524 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:47.958723 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:47.958803 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:47.958973 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:45:47.959074 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:45:47.959174 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:47.959190 (+ 16us) server_negotiation.cc:299] Negotiation successful
0928 04:45:47.959212 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 04:45:47.961238 16610 raft_consensus.cc:1261] T 1caf2c2be98d4df5b802e2789cddca73 P e6d19702051c4ea8bc3c0319416ef50e [term 1 FOLLOWER]: Refusing update from remote peer bdbad0e08ce14c83ab0de7783bdddd09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:45:47.961295 16340 raft_consensus.cc:1261] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 1 FOLLOWER]: Refusing update from remote peer bdbad0e08ce14c83ab0de7783bdddd09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:45:47.961688 16673 consensus_queue.cc:1035] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:45:47.961858 16671 consensus_queue.cc:1035] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:45:47.964139 16675 mvcc.cc:205] Tried to move back new op lower bound from 7334177783646248960 to 7334177783535251456. Current Snapshot: MvccSnapshot[applied={T|T < 7334177783646248960}]
I20260928 04:45:47.964447 16676 mvcc.cc:205] Tried to move back new op lower bound from 7334177783646248960 to 7334177783535251456. Current Snapshot: MvccSnapshot[applied={T|T < 7334177783646248960}]
I20260928 04:45:47.964576 16678 mvcc.cc:205] Tried to move back new op lower bound from 7334177783646248960 to 7334177783535251456. Current Snapshot: MvccSnapshot[applied={T|T < 7334177783646248960}]
I20260928 04:45:47.967027 16197 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:36408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:45:47.967113 16197 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:36408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:45:47.967808 16197 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c: Sending DeleteTablet for 3 replicas of tablet 1caf2c2be98d4df5b802e2789cddca73
I20260928 04:45:47.968219 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 16236
I20260928 04:45:47.968204 16458 tablet_service.cc:1514] Processing DeleteTablet for tablet 1caf2c2be98d4df5b802e2789cddca73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:45:47 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:44368
I20260928 04:45:47.968250 16590 tablet_service.cc:1514] Processing DeleteTablet for tablet 1caf2c2be98d4df5b802e2789cddca73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:45:47 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:57230
I20260928 04:45:47.968777 16685 tablet_replica.cc:330] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09: stopping tablet replica
I20260928 04:45:47.968873 16685 raft_consensus.cc:2229] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:45:47.968995 16685 raft_consensus.cc:2258] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:45:47.972711 16685 ts_tablet_manager.cc:1905] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:45:47.973008 16686 tablet_replica.cc:330] T 1caf2c2be98d4df5b802e2789cddca73 P e6d19702051c4ea8bc3c0319416ef50e: stopping tablet replica
I20260928 04:45:47.973167 16686 raft_consensus.cc:2229] T 1caf2c2be98d4df5b802e2789cddca73 P e6d19702051c4ea8bc3c0319416ef50e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:45:47.973353 16686 raft_consensus.cc:2258] T 1caf2c2be98d4df5b802e2789cddca73 P e6d19702051c4ea8bc3c0319416ef50e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:45:47.973866 16686 ts_tablet_manager.cc:1905] T 1caf2c2be98d4df5b802e2789cddca73 P e6d19702051c4ea8bc3c0319416ef50e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 04:45:47.974188 16182 connection.cc:535] client connection to 127.1.226.193:43663 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 04:45:47.974292 16182 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 04:45:47.974711 16685 ts_tablet_manager.cc:1918] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:45:47.974792 16685 log.cc:1196] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/wal/wals/1caf2c2be98d4df5b802e2789cddca73
I20260928 04:45:47.975091 16685 ts_tablet_manager.cc:1939] T 1caf2c2be98d4df5b802e2789cddca73 P bdbad0e08ce14c83ab0de7783bdddd09: Deleting consensus metadata
I20260928 04:45:47.975440 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 16391
I20260928 04:45:47.975512 16182 catalog_manager.cc:4841] TS bdbad0e08ce14c83ab0de7783bdddd09 (127.1.226.194:32989): tablet 1caf2c2be98d4df5b802e2789cddca73 (table test-table [id=bb95af94d7494381a353127bcb0f0416]) successfully deleted
I20260928 04:45:47.977680 16686 ts_tablet_manager.cc:1918] T 1caf2c2be98d4df5b802e2789cddca73 P e6d19702051c4ea8bc3c0319416ef50e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:45:47.977767 16686 log.cc:1196] T 1caf2c2be98d4df5b802e2789cddca73 P e6d19702051c4ea8bc3c0319416ef50e: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/wal/wals/1caf2c2be98d4df5b802e2789cddca73
I20260928 04:45:47.977874 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:47.977674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:43663 (local address 127.0.0.1:49418)
0928 04:45:47.977779 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:47.977835 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.193:43663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 04:45:47.978065 16686 ts_tablet_manager.cc:1939] T 1caf2c2be98d4df5b802e2789cddca73 P e6d19702051c4ea8bc3c0319416ef50e: Deleting consensus metadata
W20260928 04:45:47.978012 16182 catalog_manager.cc:4584] TS 3cfb71b57e6648cbb0786b6250f3e5c9 (127.1.226.193:43663): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1caf2c2be98d4df5b802e2789cddca73: Network error: Client connection negotiation failed: client connection to 127.1.226.193:43663: connect: Connection refused (error 111)
I20260928 04:45:47.978686 16182 catalog_manager.cc:4841] TS e6d19702051c4ea8bc3c0319416ef50e (127.1.226.195:39013): tablet 1caf2c2be98d4df5b802e2789cddca73 (table test-table [id=bb95af94d7494381a353127bcb0f0416]) successfully deleted
I20260928 04:45:47.981542 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 16527
I20260928 04:45:47.986589 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 16165
I20260928 04:45:47.991865 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:36905
--webserver_interface=127.1.226.254
--webserver_port=40761
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:36905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:45:48.075905 16688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:45:48.076149 16688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:45:48.076203 16688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:45:48.077368 16688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:45:48.077440 16688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:45:48.077464 16688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:45:48.077495 16688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:45:48.077543 16688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:45:48.079017 16688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:36905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:36905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=40761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:45:48.079298 16688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:45:48.079524 16688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:45:48.084597 16696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:48.084590 16694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:48.084591 16693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:48.084858 16688 server_base.cc:1034] running on GCE node
I20260928 04:45:48.085095 16688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:45:48.085391 16688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:45:48.086544 16688 hybrid_clock.cc:648] HybridClock initialized: now 1790570748086529 us; error 28 us; skew 500 ppm
Sep 28 04:45:48 dist-test-slave-j5c3 krb5kdc[15618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570748, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:45:48.089147 16688 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:45:48.089622 16688 webserver.cc:460] Webserver started at http://127.1.226.254:40761/ using document root <none> and password file <none>
I20260928 04:45:48.089877 16688 fs_manager.cc:359] Metadata directory not provided
I20260928 04:45:48.089948 16688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:45:48.091334 16688 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:45:48.092028 16703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:48.092237 16688 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:45:48.092338 16688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "1a0ae1415dd94dc483884d84a07bcb1c"
format_stamp: "Formatted at 2026-09-28 04:45:46 on dist-test-slave-j5c3"
server_key: "dec1fd3266d6b46b150360f071114e19"
server_key_iv: "fda9c95cbeddf048fe88bab4893da9d9"
server_key_version: "encryptionkey@0"
I20260928 04:45:48.092705 16688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:45:48.105643 16688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:45:48.107752 16688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:45:48.107933 16688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:45:48.112004 16688 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:36905
I20260928 04:45:48.112079 16755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:36905 every 8 connection(s)
I20260928 04:45:48.112504 16688 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:45:48.114125 16756 sys_catalog.cc:263] Verifying existing consensus state
I20260928 04:45:48.114789 16756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c: Bootstrap starting.
I20260928 04:45:48.116837 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 16688
I20260928 04:45:48.117164 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:43663
--local_ip_for_outbound_sockets=127.1.226.193
--tserver_master_addrs=127.1.226.254:36905
--webserver_port=38635
--webserver_interface=127.1.226.193
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:45:48.124140 16756 log.cc:824] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c: Log is configured to *not* fsync() on all Append() calls
I20260928 04:45:48.126552 16756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:45:48.126892 16756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c: Bootstrap complete.
I20260928 04:45:48.129281 16756 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0ae1415dd94dc483884d84a07bcb1c" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 36905 } }
I20260928 04:45:48.129552 16756 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a0ae1415dd94dc483884d84a07bcb1c, State: Initialized, Role: FOLLOWER
I20260928 04:45:48.129767 16756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0ae1415dd94dc483884d84a07bcb1c" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 36905 } }
I20260928 04:45:48.129866 16756 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:45:48.129920 16756 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:45:48.129967 16756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [term 2 FOLLOWER]: Advancing to term 3
I20260928 04:45:48.130951 16756 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0ae1415dd94dc483884d84a07bcb1c" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 36905 } }
I20260928 04:45:48.131071 16756 leader_election.cc:302] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a0ae1415dd94dc483884d84a07bcb1c; no voters:
I20260928 04:45:48.131269 16756 leader_election.cc:288] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 04:45:48.131306 16760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [term 3 FOLLOWER]: Leader election won for term 3
I20260928 04:45:48.131409 16760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [term 3 LEADER]: Becoming Leader. State: Replica: 1a0ae1415dd94dc483884d84a07bcb1c, State: Running, Role: LEADER
I20260928 04:45:48.131511 16760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0ae1415dd94dc483884d84a07bcb1c" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 36905 } }
I20260928 04:45:48.131557 16756 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:45:48.131793 16761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1a0ae1415dd94dc483884d84a07bcb1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0ae1415dd94dc483884d84a07bcb1c" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 36905 } } }
I20260928 04:45:48.131897 16761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [sys.catalog]: This master's current role is: LEADER
I20260928 04:45:48.132087 16762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a0ae1415dd94dc483884d84a07bcb1c. Latest consensus state: current_term: 3 leader_uuid: "1a0ae1415dd94dc483884d84a07bcb1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0ae1415dd94dc483884d84a07bcb1c" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 36905 } } }
I20260928 04:45:48.132179 16762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a0ae1415dd94dc483884d84a07bcb1c [sys.catalog]: This master's current role is: LEADER
I20260928 04:45:48.132369 16765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:45:48.133188 16765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:45:48.133620 16765 catalog_manager.cc:1259] Loaded cluster ID: c8e908171bd24a1ca57f81a48a3d9fb5
I20260928 04:45:48.133663 16765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 04:45:48.133980 16765 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1e2ec779ad google::LogMessage::Fail() at ??:0
@ 0x7f1e2ec7a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f1e2ec774de google::LogMessage::Flush() at ??:0
@ 0x7f1e2ec78a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1e36b980f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f1e36ba18b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1e30b450d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1e30b3a954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1e3199e6db start_thread at ??:0
@ 0x7f1e2d77561f clone at ??:0
W20260928 04:45:48.218110 16758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:45:48.218333 16758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:45:48.218360 16758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:45:48.219615 16758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:45:48.219667 16758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:45:48.219703 16758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:45:48.222205 16758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:43663
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=38635
--tserver_master_addrs=127.1.226.254:36905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:45:48.222512 16758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:45:48.222805 16758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:45:48.230971 16793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:48.231474 16795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:48.231688 16792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:48.232228 16758 server_base.cc:1034] running on GCE node
I20260928 04:45:48.232481 16758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:45:48.232815 16758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:45:48.233992 16758 hybrid_clock.cc:648] HybridClock initialized: now 1790570748233961 us; error 54 us; skew 500 ppm
Sep 28 04:45:48 dist-test-slave-j5c3 krb5kdc[15618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570748, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:45:48.237373 16758 init.cc:377] Logged in from keytab as kudu/127.1.226.193@KRBTEST.COM (short username kudu)
I20260928 04:45:48.237953 16758 webserver.cc:460] Webserver started at http://127.1.226.193:38635/ using document root <none> and password file <none>
I20260928 04:45:48.238258 16758 fs_manager.cc:359] Metadata directory not provided
I20260928 04:45:48.238335 16758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:45:48.240545 16758 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:45:48.241525 16802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:48.241758 16758 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:45:48.241873 16758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "3cfb71b57e6648cbb0786b6250f3e5c9"
format_stamp: "Formatted at 2026-09-28 04:45:46 on dist-test-slave-j5c3"
server_key: "0833c0510044e057eb27d0b290dd40e8"
server_key_iv: "263e9b97f9e55a381c914d87c669cb6b"
server_key_version: "encryptionkey@0"
I20260928 04:45:48.242393 16758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:45:48.254705 16758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:45:48.257184 16758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:45:48.257403 16758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:45:48.257730 16758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:45:48.258253 16809 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 04:45:48.260264 16758 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 04:45:48.260339 16758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 04:45:48.260386 16758 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 04:45:48.261096 16758 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 04:45:48.261152 16758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 04:45:48.261245 16809 tablet_bootstrap.cc:492] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9: Bootstrap starting.
I20260928 04:45:48.268514 16758 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:43663
I20260928 04:45:48.268909 16758 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
I20260928 04:45:48.269544 16916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:43663 every 8 connection(s)
I20260928 04:45:48.272991 16809 log.cc:824] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9: Log is configured to *not* fsync() on all Append() calls
I20260928 04:45:48.273885 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 16758
I20260928 04:45:48.274170 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:32989
--local_ip_for_outbound_sockets=127.1.226.194
--tserver_master_addrs=127.1.226.254:36905
--webserver_port=42245
--webserver_interface=127.1.226.194
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:45:48.275835 16809 tablet_bootstrap.cc:492] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:45:48.276209 16809 tablet_bootstrap.cc:492] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9: Bootstrap complete.
I20260928 04:45:48.277432 16809 ts_tablet_manager.cc:1397] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.011s
I20260928 04:45:48.278782 16809 raft_consensus.cc:348] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32989 } } peers { permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 } } peers { permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 } }
I20260928 04:45:48.279107 16809 raft_consensus.cc:729] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cfb71b57e6648cbb0786b6250f3e5c9, State: Initialized, Role: FOLLOWER
I20260928 04:45:48.279273 16809 consensus_queue.cc:260] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32989 } } peers { permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 } } peers { permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 } }
I20260928 04:45:48.279577 16809 ts_tablet_manager.cc:1428] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
W20260928 04:45:48.357635 16921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:45:48.357892 16921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:45:48.357929 16921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:45:48.359777 16921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:45:48.359850 16921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:45:48.359896 16921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:45:48.361459 16921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:32989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=42245
--tserver_master_addrs=127.1.226.254:36905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:45:48.361733 16921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:45:48.361970 16921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:45:48.362627 16919 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:45:48.269653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.193:51527)
0928 04:45:48.269849 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:48.269894 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:48.270364 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:48.362412 (+ 92048us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":113,"threads_started":1}
W20260928 04:45:48.362886 16917 heartbeater.cc:644] Failed to heartbeat to 127.1.226.254:36905 (0 consecutive failures): Network error: Failed to ping master at 127.1.226.254:36905: Client connection negotiation failed: client connection to 127.1.226.254:36905: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 04:45:48.363271 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:48.363136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.193:47451)
0928 04:45:48.363193 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:48.363244 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 04:45:48.363685 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:48.363589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.193:56625)
0928 04:45:48.363634 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:48.363656 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 04:45:48.363799 16917 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 04:45:48.367550 16927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:48.367594 16930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:48.367697 16921 server_base.cc:1034] running on GCE node
W20260928 04:45:48.367564 16928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:48.367961 16921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:45:48.368248 16921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:45:48.369403 16921 hybrid_clock.cc:648] HybridClock initialized: now 1790570748369385 us; error 39 us; skew 500 ppm
Sep 28 04:45:48 dist-test-slave-j5c3 krb5kdc[15618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570748, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:45:48.371922 16921 init.cc:377] Logged in from keytab as kudu/127.1.226.194@KRBTEST.COM (short username kudu)
I20260928 04:45:48.372408 16921 webserver.cc:460] Webserver started at http://127.1.226.194:42245/ using document root <none> and password file <none>
I20260928 04:45:48.372596 16921 fs_manager.cc:359] Metadata directory not provided
I20260928 04:45:48.372635 16921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:45:48.373916 16921 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:45:48.374469 16937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:48.374644 16921 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:45:48.374709 16921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "bdbad0e08ce14c83ab0de7783bdddd09"
format_stamp: "Formatted at 2026-09-28 04:45:46 on dist-test-slave-j5c3"
server_key: "9c539262f6d69ee1539b64df0cb6e0fc"
server_key_iv: "58630404ba18b4b6605f8146f62c191e"
server_key_version: "encryptionkey@0"
I20260928 04:45:48.375025 16921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:45:48.383641 16921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:45:48.385905 16921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:45:48.386014 16921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:45:48.386184 16921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:45:48.386420 16921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:45:48.386449 16921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:48.386471 16921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:45:48.386483 16921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:48.392379 16921 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:32989
I20260928 04:45:48.392433 17050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:32989 every 8 connection(s)
I20260928 04:45:48.392766 16921 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
I20260928 04:45:48.393826 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:48.393410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.194:44673)
0928 04:45:48.393575 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:48.393650 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
W20260928 04:45:48.394215 17051 heartbeater.cc:644] Failed to heartbeat to 127.1.226.254:36905 (0 consecutive failures): Network error: Failed to ping master at 127.1.226.254:36905: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
I20260928 04:45:48.394543 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:48.394445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.194:52433)
0928 04:45:48.394496 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:48.394525 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 04:45:48.394933 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:48.394797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.194:51827)
0928 04:45:48.394875 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:48.394906 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 04:45:48.395121 17051 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 04:45:48.399582 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 16921
I20260928 04:45:48.399809 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:39013
--local_ip_for_outbound_sockets=127.1.226.195
--tserver_master_addrs=127.1.226.254:36905
--webserver_port=44139
--webserver_interface=127.1.226.195
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:45:48.484397 17054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:45:48.484637 17054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:45:48.484689 17054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:45:48.486011 17054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:45:48.486085 17054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:45:48.486140 17054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:45:48.489064 17054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:39013
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=44139
--tserver_master_addrs=127.1.226.254:36905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:45:48.489408 17054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:45:48.489681 17054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:45:48.495344 17060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:45:48.495307 17059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:48.495533 17054 server_base.cc:1034] running on GCE node
W20260928 04:45:48.495303 17062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:45:48.495795 17054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:45:48.496052 17054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:45:48.497200 17054 hybrid_clock.cc:648] HybridClock initialized: now 1790570748497198 us; error 39 us; skew 500 ppm
Sep 28 04:45:48 dist-test-slave-j5c3 krb5kdc[15618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570748, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:45:48.499990 17054 init.cc:377] Logged in from keytab as kudu/127.1.226.195@KRBTEST.COM (short username kudu)
I20260928 04:45:48.500661 17054 webserver.cc:460] Webserver started at http://127.1.226.195:44139/ using document root <none> and password file <none>
I20260928 04:45:48.500897 17054 fs_manager.cc:359] Metadata directory not provided
I20260928 04:45:48.500948 17054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:45:48.502269 17054 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:45:48.502830 17069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:48.503010 17054 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:45:48.503118 17054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "e6d19702051c4ea8bc3c0319416ef50e"
format_stamp: "Formatted at 2026-09-28 04:45:47 on dist-test-slave-j5c3"
server_key: "48bb458b125d07faccd8ca28feb9d2e2"
server_key_iv: "0c219606240e6d95f82d8667ade36b6e"
server_key_version: "encryptionkey@0"
I20260928 04:45:48.503485 17054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:45:48.519729 17054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:45:48.522297 17054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:45:48.522467 17054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:45:48.522715 17054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:45:48.523092 17054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:45:48.523146 17054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:48.523190 17054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:45:48.523216 17054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:45:48.529207 17054 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:39013
I20260928 04:45:48.529251 17182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:39013 every 8 connection(s)
I20260928 04:45:48.529651 17054 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
I20260928 04:45:48.530768 17185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:48.530387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.195:51703)
0928 04:45:48.530549 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:48.530623 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
W20260928 04:45:48.531194 17183 heartbeater.cc:644] Failed to heartbeat to 127.1.226.254:36905 (0 consecutive failures): Network error: Failed to ping master at 127.1.226.254:36905: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
I20260928 04:45:48.531481 17185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:48.531384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.195:55489)
0928 04:45:48.531434 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:48.531458 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 04:45:48.531759 17185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:48.531676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.195:36271)
0928 04:45:48.531718 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:48.531739 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 04:45:48.531873 17183 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 04:45:48.535068 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 17054
I20260928 04:45:49.364449 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:49.364141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.193:56029)
0928 04:45:49.364340 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:49.364382 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":114,"threads_started":1}
I20260928 04:45:49.395730 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:49.395491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.194:45089)
0928 04:45:49.395632 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:49.395669 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":61,"threads_started":1}
I20260928 04:45:49.532521 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:49.532311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.195:44811)
0928 04:45:49.532429 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:49.532468 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260928 04:45:49.773564 17190 raft_consensus.cc:482] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:45:49.773749 17190 raft_consensus.cc:504] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32989 } } peers { permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 } } peers { permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 } }
I20260928 04:45:49.774097 17190 leader_election.cc:288] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bdbad0e08ce14c83ab0de7783bdddd09 (127.1.226.194:32989), e6d19702051c4ea8bc3c0319416ef50e (127.1.226.195:39013)
Sep 28 04:45:49 dist-test-slave-j5c3 krb5kdc[15618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570748, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for kudu/127.1.226.194@KRBTEST.COM
Sep 28 04:45:49 dist-test-slave-j5c3 krb5kdc[15618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570748, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for kudu/127.1.226.195@KRBTEST.COM
I20260928 04:45:49.785243 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:49.774327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:32989 (local address 127.1.226.193:42633)
0928 04:45:49.774410 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:49.774426 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:49.774511 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:49.775572 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:49.775575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:49.775932 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:45:49.776132 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:49.776138 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:49.777280 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:49.777281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:49.778209 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:49.778211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:49.778939 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:49.778940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:49.779065 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:49.779703 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:45:49.779712 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:45:49.781372 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:45:49.783550 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:49.783559 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:49.783562 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:49.783864 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:49.784128 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:49.784129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:49.784129 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:49.784273 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:49.784694 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:45:49.784695 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:45:49.784819 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 04:45:49.784927 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 04:45:49.785153 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":35}
I20260928 04:45:49.785272 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:49.774461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:42633 (local address 127.1.226.194:32989)
0928 04:45:49.774641 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:49.774655 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:49.774692 (+ 37us) server_negotiation.cc:407] Connection header received
0928 04:45:49.775297 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:49.775300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:49.775362 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:49.775514 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:49.776300 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:49.777170 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:49.778302 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:49.778834 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:49.778960 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:49.781508 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:49.781514 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:49.781515 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:49.781530 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:49.783454 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:49.783947 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:49.783951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:49.783952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:49.784059 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:49.784345 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:49.784347 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:49.784347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:49.784605 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:49.784699 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:49.784998 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 04:45:49.785127 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":69,"threads_started":1}
I20260928 04:45:49.785602 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:49.774479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:39013 (local address 127.1.226.193:34923)
0928 04:45:49.774610 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:49.774619 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:45:49.774660 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:45:49.775645 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:45:49.775647 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:45:49.775936 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:45:49.776132 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:49.776138 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:49.777301 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:49.777302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:49.778208 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:45:49.778211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:45:49.778937 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:45:49.778938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:45:49.779064 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:49.779747 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:45:49.779752 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:45:49.781736 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:45:49.783644 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:49.783645 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:49.783645 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:49.783887 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:49.784215 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:45:49.784216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:45:49.784216 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:45:49.784281 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:45:49.784682 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:45:49.784689 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:45:49.784819 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 04:45:49.784929 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 04:45:49.785239 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":190,"thread_start_us":62,"threads_started":1}
I20260928 04:45:49.786070 17193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:49.774573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:34923 (local address 127.1.226.195:39013)
0928 04:45:49.774727 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 04:45:49.774734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 04:45:49.774758 (+ 24us) server_negotiation.cc:407] Connection header received
0928 04:45:49.775297 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:45:49.775300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:45:49.775362 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:45:49.775513 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:45:49.776300 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:49.777173 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:49.778306 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:45:49.778809 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:45:49.778955 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:45:49.781866 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:45:49.781870 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:45:49.781871 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:45:49.781883 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:45:49.783549 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:49.784002 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:49.784004 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:49.784005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:49.784115 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:45:49.784368 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:45:49.784369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:45:49.784369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:45:49.784605 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:45:49.784700 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 04:45:49.784991 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 04:45:49.785123 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
W20260928 04:45:49.786161 16803 leader_election.cc:341] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bdbad0e08ce14c83ab0de7783bdddd09 (127.1.226.194:32989): Not found: Tablet not found: 1caf2c2be98d4df5b802e2789cddca73
W20260928 04:45:49.786758 16803 leader_election.cc:341] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e6d19702051c4ea8bc3c0319416ef50e (127.1.226.195:39013): Not found: Tablet not found: 1caf2c2be98d4df5b802e2789cddca73
I20260928 04:45:49.786806 16803 leader_election.cc:302] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cfb71b57e6648cbb0786b6250f3e5c9; no voters: bdbad0e08ce14c83ab0de7783bdddd09, e6d19702051c4ea8bc3c0319416ef50e
I20260928 04:45:49.786949 17190 raft_consensus.cc:2726] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 04:45:50.365186 17194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:50.364953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.193:38419)
0928 04:45:50.365091 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:50.365138 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":60,"threads_started":1}
I20260928 04:45:50.396559 17195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:50.396342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.194:36377)
0928 04:45:50.396471 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:50.396517 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
I20260928 04:45:50.533218 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:50.532988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.195:34367)
0928 04:45:50.533131 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:50.533177 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":63,"threads_started":1}
I20260928 04:45:51.365937 17197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:51.365684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.193:59239)
0928 04:45:51.365830 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:51.365876 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":66,"threads_started":1}
I20260928 04:45:51.397401 17198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:51.397195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.194:42075)
0928 04:45:51.397319 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:51.397361 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260928 04:45:51.533957 17199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:51.533730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.195:33561)
0928 04:45:51.533876 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:51.533916 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":72,"threads_started":1}
I20260928 04:45:51.998685 17200 raft_consensus.cc:482] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:45:51.998785 17200 raft_consensus.cc:504] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32989 } } peers { permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 } } peers { permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 } }
I20260928 04:45:51.998951 17200 leader_election.cc:288] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bdbad0e08ce14c83ab0de7783bdddd09 (127.1.226.194:32989), e6d19702051c4ea8bc3c0319416ef50e (127.1.226.195:39013)
W20260928 04:45:51.999451 16803 leader_election.cc:341] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e6d19702051c4ea8bc3c0319416ef50e (127.1.226.195:39013): Not found: Tablet not found: 1caf2c2be98d4df5b802e2789cddca73
W20260928 04:45:51.999527 16803 leader_election.cc:341] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bdbad0e08ce14c83ab0de7783bdddd09 (127.1.226.194:32989): Not found: Tablet not found: 1caf2c2be98d4df5b802e2789cddca73
I20260928 04:45:51.999575 16803 leader_election.cc:302] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cfb71b57e6648cbb0786b6250f3e5c9; no voters: bdbad0e08ce14c83ab0de7783bdddd09, e6d19702051c4ea8bc3c0319416ef50e
I20260928 04:45:51.999650 17200 raft_consensus.cc:2726] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 04:45:52.366760 17201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:52.366491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.193:48065)
0928 04:45:52.366640 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:52.366686 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":66,"threads_started":1}
I20260928 04:45:52.398154 17202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:52.397921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.194:53815)
0928 04:45:52.398061 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:52.398095 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
I20260928 04:45:52.534734 17203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:52.534512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.195:40867)
0928 04:45:52.534652 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:52.534692 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":72,"threads_started":1}
I20260928 04:45:53.367645 17204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:53.367353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.193:34843)
0928 04:45:53.367548 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:53.367597 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":82,"threads_started":1}
I20260928 04:45:53.398931 17205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:53.398714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.194:54477)
0928 04:45:53.398847 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:53.398887 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260928 04:45:53.535531 17206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:53.535312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.195:47469)
0928 04:45:53.535440 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:53.535483 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
I20260928 04:45:54.368490 17207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:54.368252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.193:53791)
0928 04:45:54.368394 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:54.368441 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
I20260928 04:45:54.399622 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:54.399413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.194:47193)
0928 04:45:54.399539 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:54.399575 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260928 04:45:54.536302 17209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:54.535955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.195:39993)
0928 04:45:54.536114 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:54.536251 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":62,"threads_started":1}
I20260928 04:45:54.627617 17210 raft_consensus.cc:482] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:45:54.627754 17210 raft_consensus.cc:504] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32989 } } peers { permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 } } peers { permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 } }
I20260928 04:45:54.627919 17210 leader_election.cc:288] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bdbad0e08ce14c83ab0de7783bdddd09 (127.1.226.194:32989), e6d19702051c4ea8bc3c0319416ef50e (127.1.226.195:39013)
W20260928 04:45:54.628383 16803 leader_election.cc:341] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e6d19702051c4ea8bc3c0319416ef50e (127.1.226.195:39013): Not found: Tablet not found: 1caf2c2be98d4df5b802e2789cddca73
W20260928 04:45:54.628468 16803 leader_election.cc:341] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bdbad0e08ce14c83ab0de7783bdddd09 (127.1.226.194:32989): Not found: Tablet not found: 1caf2c2be98d4df5b802e2789cddca73
I20260928 04:45:54.628485 16803 leader_election.cc:302] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cfb71b57e6648cbb0786b6250f3e5c9; no voters: bdbad0e08ce14c83ab0de7783bdddd09, e6d19702051c4ea8bc3c0319416ef50e
I20260928 04:45:54.628556 17210 raft_consensus.cc:2726] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 04:45:55.369349 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:55.369089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.193:38089)
0928 04:45:55.369247 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:55.369300 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":68,"threads_started":1}
I20260928 04:45:55.400449 17212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:55.400146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.194:51805)
0928 04:45:55.400361 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:55.400403 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":101,"threads_started":1}
I20260928 04:45:55.537092 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:55.536846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.195:39691)
0928 04:45:55.536994 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:55.537037 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
I20260928 04:45:56.370223 17214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:56.369961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.193:42889)
0928 04:45:56.370119 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:56.370167 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":67,"threads_started":1}
I20260928 04:45:56.401242 17215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:56.400977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.194:47399)
0928 04:45:56.401153 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:56.401196 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":80,"threads_started":1}
I20260928 04:45:56.537879 17216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:56.537635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.195:48717)
0928 04:45:56.537782 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:56.537832 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20260928 04:45:57.371032 17217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:57.370751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.193:57983)
0928 04:45:57.370933 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:57.370974 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":104,"threads_started":1}
I20260928 04:45:57.401985 17218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:57.401754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.194:57623)
0928 04:45:57.401899 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:57.401937 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260928 04:45:57.538661 17219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:57.538423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.195:60495)
0928 04:45:57.538569 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:57.538612 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
I20260928 04:45:58.371747 17220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:58.371494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.193:55497)
0928 04:45:58.371644 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:58.371698 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":64,"threads_started":1}
I20260928 04:45:58.402626 17221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:58.402399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.194:43789)
0928 04:45:58.402539 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:58.402571 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260928 04:45:58.539491 17222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:58.539236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.195:58715)
0928 04:45:58.539394 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:58.539425 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
I20260928 04:45:58.630148 17223 raft_consensus.cc:482] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:45:58.630302 17223 raft_consensus.cc:504] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32989 } } peers { permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 } } peers { permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 } }
I20260928 04:45:58.630502 17223 leader_election.cc:288] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bdbad0e08ce14c83ab0de7783bdddd09 (127.1.226.194:32989), e6d19702051c4ea8bc3c0319416ef50e (127.1.226.195:39013)
W20260928 04:45:58.630978 16803 leader_election.cc:341] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bdbad0e08ce14c83ab0de7783bdddd09 (127.1.226.194:32989): Not found: Tablet not found: 1caf2c2be98d4df5b802e2789cddca73
W20260928 04:45:58.631047 16803 leader_election.cc:341] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e6d19702051c4ea8bc3c0319416ef50e (127.1.226.195:39013): Not found: Tablet not found: 1caf2c2be98d4df5b802e2789cddca73
I20260928 04:45:58.631064 16803 leader_election.cc:302] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cfb71b57e6648cbb0786b6250f3e5c9; no voters: bdbad0e08ce14c83ab0de7783bdddd09, e6d19702051c4ea8bc3c0319416ef50e
I20260928 04:45:58.631127 17223 raft_consensus.cc:2726] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 04:45:59.372633 17224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:59.372384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.193:40825)
0928 04:45:59.372535 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:59.372586 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":73,"threads_started":1}
I20260928 04:45:59.403416 17225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:59.403182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.194:59747)
0928 04:45:59.403326 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:59.403364 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
I20260928 04:45:59.540345 17226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:45:59.540015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.195:33171)
0928 04:45:59.540240 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 04:45:59.540294 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":84,"threads_started":1}
I20260928 04:46:00.373497 17227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:00.373171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.193:45665)
0928 04:46:00.373388 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:00.373443 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":85,"threads_started":1}
I20260928 04:46:00.404130 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:00.403894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.194:53573)
0928 04:46:00.404047 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:00.404081 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
I20260928 04:46:00.541177 17229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:00.540926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.195:43011)
0928 04:46:00.541086 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:00.541127 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":77,"threads_started":1}
I20260928 04:46:01.356864 17230 raft_consensus.cc:482] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:46:01.356984 17230 raft_consensus.cc:504] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdbad0e08ce14c83ab0de7783bdddd09" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 32989 } } peers { permanent_uuid: "e6d19702051c4ea8bc3c0319416ef50e" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 39013 } } peers { permanent_uuid: "3cfb71b57e6648cbb0786b6250f3e5c9" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43663 } }
I20260928 04:46:01.357151 17230 leader_election.cc:288] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bdbad0e08ce14c83ab0de7783bdddd09 (127.1.226.194:32989), e6d19702051c4ea8bc3c0319416ef50e (127.1.226.195:39013)
W20260928 04:46:01.357569 16803 leader_election.cc:341] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bdbad0e08ce14c83ab0de7783bdddd09 (127.1.226.194:32989): Not found: Tablet not found: 1caf2c2be98d4df5b802e2789cddca73
W20260928 04:46:01.357649 16803 leader_election.cc:341] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e6d19702051c4ea8bc3c0319416ef50e (127.1.226.195:39013): Not found: Tablet not found: 1caf2c2be98d4df5b802e2789cddca73
I20260928 04:46:01.357666 16803 leader_election.cc:302] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cfb71b57e6648cbb0786b6250f3e5c9; no voters: bdbad0e08ce14c83ab0de7783bdddd09, e6d19702051c4ea8bc3c0319416ef50e
I20260928 04:46:01.357738 17230 raft_consensus.cc:2726] T 1caf2c2be98d4df5b802e2789cddca73 P 3cfb71b57e6648cbb0786b6250f3e5c9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 04:46:01.374359 17231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:01.374117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.193:43031)
0928 04:46:01.374271 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:01.374311 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":74,"threads_started":1}
I20260928 04:46:01.404963 17232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:01.404702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.194:38991)
0928 04:46:01.404868 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:01.404901 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":77,"threads_started":1}
I20260928 04:46:01.541913 17233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:01.541673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.195:59393)
0928 04:46:01.541814 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:01.541862 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":61,"threads_started":1}
I20260928 04:46:02.375124 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:02.374899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.193:33957)
0928 04:46:02.375035 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:02.375079 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260928 04:46:02.405748 17235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:02.405438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.194:45345)
0928 04:46:02.405588 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:02.405634 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":61,"threads_started":1}
I20260928 04:46:02.542672 17236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:02.542425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.195:43487)
0928 04:46:02.542569 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:02.542624 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":62,"threads_started":1}
I20260928 04:46:03.375851 17237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:03.375645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.193:37883)
0928 04:46:03.375769 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:03.375804 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260928 04:46:03.406610 17238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:03.406379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.194:58345)
0928 04:46:03.406524 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:03.406561 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
I20260928 04:46:03.539284 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 16758
I20260928 04:46:03.543466 17239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:03.543255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:36905 (local address 127.1.226.195:60199)
0928 04:46:03.543379 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:03.543421 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260928 04:46:03.544427 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 16921
I20260928 04:46:03.548961 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 17054
2026-09-28T04:46:03Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18252 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:46:03 dist-test-slave-j5c3 krb5kdc[17242](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:46:03 dist-test-slave-j5c3 krb5kdc[17242](info): set up 2 sockets
Sep 28 04:46:03 dist-test-slave-j5c3 krb5kdc[17242](info): commencing operation
krb5kdc: starting...
W20260928 04:46:04.612860 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:46:04 dist-test-slave-j5c3 krb5kdc[17242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:46:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:46:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:46:04.736097 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:44913
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:44913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:46:04.821947 17255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:46:04.822202 17255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:46:04.822239 17255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:46:04.823413 17255 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 04:46:04.823482 17255 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 04:46:04.823503 17255 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 04:46:04.823523 17255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:46:04.823553 17255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:46:04.823585 17255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:46:04.823608 17255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:46:04.823637 17255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:46:04.825138 17255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:44913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:44913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:46:04.825421 17255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:46:04.825654 17255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:46:04.830600 17263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:46:04.830608 17260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:46:04.830608 17261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:46:04.830767 17255 server_base.cc:1034] running on GCE node
I20260928 04:46:04.831075 17255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:46:04.831368 17255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:46:04.832564 17255 hybrid_clock.cc:648] HybridClock initialized: now 1790570764832542 us; error 43 us; skew 500 ppm
Sep 28 04:46:04 dist-test-slave-j5c3 krb5kdc[17242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570764, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:46:04.835112 17255 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:46:04.835578 17255 webserver.cc:460] Webserver started at http://127.1.226.254:45277/ using document root <none> and password file <none>
I20260928 04:46:04.835762 17255 fs_manager.cc:359] Metadata directory not provided
I20260928 04:46:04.835810 17255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:46:04.835894 17255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:46:04.837028 17255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "7de1202fbe4b4f80bdb46a2663c75504"
format_stamp: "Formatted at 2026-09-28 04:46:04 on dist-test-slave-j5c3"
server_key: "cb44743a5d0eed75b4d08779cfd9454a"
server_key_iv: "b4cd6b9ca4fceb692159fcaac1aa626f"
server_key_version: "encryptionkey@0"
I20260928 04:46:04.837329 17255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "7de1202fbe4b4f80bdb46a2663c75504"
format_stamp: "Formatted at 2026-09-28 04:46:04 on dist-test-slave-j5c3"
server_key: "cb44743a5d0eed75b4d08779cfd9454a"
server_key_iv: "b4cd6b9ca4fceb692159fcaac1aa626f"
server_key_version: "encryptionkey@0"
I20260928 04:46:04.838646 17255 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:46:04.839488 17270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:46:04.839710 17255 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:46:04.839818 17255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "7de1202fbe4b4f80bdb46a2663c75504"
format_stamp: "Formatted at 2026-09-28 04:46:04 on dist-test-slave-j5c3"
server_key: "cb44743a5d0eed75b4d08779cfd9454a"
server_key_iv: "b4cd6b9ca4fceb692159fcaac1aa626f"
server_key_version: "encryptionkey@0"
I20260928 04:46:04.839881 17255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:46:04.853490 17255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:46:04.855540 17255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:46:04.855662 17255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:46:04.859760 17255 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:44913
I20260928 04:46:04.859802 17322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:44913 every 8 connection(s)
I20260928 04:46:04.860200 17255 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:46:04.861141 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 17255
I20260928 04:46:04.861196 17323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:46:04.861258 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:46:04.861485 1931 external_mini_cluster.cc:1428] Setting key e16e5e107724c75f9efaad53e5f36f60
I20260928 04:46:04.864718 17323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504: Bootstrap starting.
I20260928 04:46:04.865572 17323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504: Neither blocks nor log segments found. Creating new log.
I20260928 04:46:04.865895 17323 log.cc:824] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504: Log is configured to *not* fsync() on all Append() calls
I20260928 04:46:04.866710 17323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504: No bootstrap required, opened a new log
Sep 28 04:46:04 dist-test-slave-j5c3 krb5kdc[17242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:46:04.868940 17323 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de1202fbe4b4f80bdb46a2663c75504" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44913 } }
I20260928 04:46:04.869105 17323 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:46:04.869151 17323 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7de1202fbe4b4f80bdb46a2663c75504, State: Initialized, Role: FOLLOWER
I20260928 04:46:04.869323 17323 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de1202fbe4b4f80bdb46a2663c75504" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44913 } }
I20260928 04:46:04.869457 17323 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:46:04.869499 17323 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:46:04.869554 17323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:46:04.870388 17323 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de1202fbe4b4f80bdb46a2663c75504" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44913 } }
I20260928 04:46:04.870500 17323 leader_election.cc:302] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7de1202fbe4b4f80bdb46a2663c75504; no voters:
I20260928 04:46:04.870739 17323 leader_election.cc:288] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:46:04.870775 17328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:46:04.870904 17328 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [term 1 LEADER]: Becoming Leader. State: Replica: 7de1202fbe4b4f80bdb46a2663c75504, State: Running, Role: LEADER
I20260928 04:46:04.871033 17323 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:46:04.871044 17328 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de1202fbe4b4f80bdb46a2663c75504" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44913 } }
I20260928 04:46:04.871505 17328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7de1202fbe4b4f80bdb46a2663c75504. Latest consensus state: current_term: 1 leader_uuid: "7de1202fbe4b4f80bdb46a2663c75504" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de1202fbe4b4f80bdb46a2663c75504" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44913 } } }
I20260928 04:46:04.871780 17328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [sys.catalog]: This master's current role is: LEADER
I20260928 04:46:04.871701 17330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7de1202fbe4b4f80bdb46a2663c75504" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de1202fbe4b4f80bdb46a2663c75504" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44913 } } }
I20260928 04:46:04.872267 17330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [sys.catalog]: This master's current role is: LEADER
I20260928 04:46:04.872392 17337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:46:04.873124 17337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:46:04.873590 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:04.862057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59216 (local address 127.1.226.254:44913)
0928 04:46:04.862222 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 04:46:04.862231 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:46:04.862256 (+ 25us) server_negotiation.cc:407] Connection header received
0928 04:46:04.862788 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:46:04.862797 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:46:04.862849 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:46:04.862989 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:46:04.863839 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:04.864872 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:04.866031 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:04.866614 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:04.866728 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:04.869303 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:46:04.869308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:46:04.869308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:46:04.869321 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:46:04.871775 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:46:04.872267 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:46:04.872270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:46:04.872271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:46:04.872408 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:46:04.872646 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:46:04.872647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:46:04.872647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:46:04.872896 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:46:04.873073 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 04:46:04.873177 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 04:46:04.873428 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20260928 04:46:04.874806 17337 catalog_manager.cc:1347] Generated new cluster ID: fa05fa53151e461ca22e53a29d32bbe1
I20260928 04:46:04.874863 17337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:46:04.880388 17337 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:46:04.880905 17337 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:46:04.884070 17337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504: Generated new TSK 0
I20260928 04:46:04.884266 17337 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.226.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
I20260928 04:46:04.911273 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:0
--local_ip_for_outbound_sockets=127.1.226.193
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:44913
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:46:04.994612 17349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:46:04.994868 17349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:46:04.994906 17349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:46:04.996112 17349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:46:04.996217 17349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:46:04.996291 17349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:46:04.997854 17349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:44913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:46:04.998138 17349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:46:04.998391 17349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:46:05.003998 17355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:46:05.004050 17354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:46:05.004195 17349 server_base.cc:1034] running on GCE node
W20260928 04:46:05.004232 17357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:46:05.004536 17349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:46:05.004802 17349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:46:05.005978 17349 hybrid_clock.cc:648] HybridClock initialized: now 1790570765005942 us; error 60 us; skew 500 ppm
Sep 28 04:46:05 dist-test-slave-j5c3 krb5kdc[17242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570765, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:46:05.009040 17349 init.cc:377] Logged in from keytab as kudu/127.1.226.193@KRBTEST.COM (short username kudu)
I20260928 04:46:05.009610 17349 webserver.cc:460] Webserver started at http://127.1.226.193:32933/ using document root <none> and password file <none>
I20260928 04:46:05.009850 17349 fs_manager.cc:359] Metadata directory not provided
I20260928 04:46:05.009897 17349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:46:05.010040 17349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:46:05.011140 17349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/data/instance:
uuid: "9c8dc7ae435f4dc4a223d2996000aeda"
format_stamp: "Formatted at 2026-09-28 04:46:05 on dist-test-slave-j5c3"
server_key: "6555c58716ad430b02df54ea4ec3b1ff"
server_key_iv: "c1e73899d9b3de377b1a75a6f446a72b"
server_key_version: "encryptionkey@0"
I20260928 04:46:05.011514 17349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance:
uuid: "9c8dc7ae435f4dc4a223d2996000aeda"
format_stamp: "Formatted at 2026-09-28 04:46:05 on dist-test-slave-j5c3"
server_key: "6555c58716ad430b02df54ea4ec3b1ff"
server_key_iv: "c1e73899d9b3de377b1a75a6f446a72b"
server_key_version: "encryptionkey@0"
I20260928 04:46:05.013079 17349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:46:05.014052 17364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:46:05.014292 17349 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:46:05.014405 17349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "9c8dc7ae435f4dc4a223d2996000aeda"
format_stamp: "Formatted at 2026-09-28 04:46:05 on dist-test-slave-j5c3"
server_key: "6555c58716ad430b02df54ea4ec3b1ff"
server_key_iv: "c1e73899d9b3de377b1a75a6f446a72b"
server_key_version: "encryptionkey@0"
I20260928 04:46:05.014488 17349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:46:05.025228 17349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:46:05.027222 17349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:46:05.027375 17349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:46:05.027612 17349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:46:05.027938 17349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:46:05.027990 17349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:46:05.028033 17349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:46:05.028059 17349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:46:05.033844 17349 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:36515
I20260928 04:46:05.033898 17477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:36515 every 8 connection(s)
I20260928 04:46:05.034243 17349 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
I20260928 04:46:05.036559 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 17349
I20260928 04:46:05.036695 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance
I20260928 04:46:05.036974 1931 external_mini_cluster.cc:1428] Setting key 4f7fefad3c87692128f57ec064e99bd5
Sep 28 04:46:05 dist-test-slave-j5c3 krb5kdc[17242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570765, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:46:05.045550 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:05.035025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:36743 (local address 127.1.226.254:44913)
0928 04:46:05.035106 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 04:46:05.035108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:46:05.035302 (+ 194us) server_negotiation.cc:407] Connection header received
0928 04:46:05.035846 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:46:05.035847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:46:05.035861 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:46:05.035924 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:46:05.037137 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.037782 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.038915 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.039263 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.039310 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:05.041880 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:46:05.041882 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:46:05.041882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:46:05.041891 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:46:05.043571 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:46:05.044098 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:46:05.044100 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:46:05.044100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:46:05.044200 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:46:05.044535 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:46:05.044536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:46:05.044536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:46:05.044732 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:46:05.044819 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 04:46:05.045364 (+ 545us) server_negotiation.cc:299] Negotiation successful
0928 04:46:05.045461 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 04:46:05.045876 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:05.034994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.193:36743)
0928 04:46:05.035169 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:05.035206 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 04:46:05.035724 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:46:05.036000 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:46:05.036003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:46:05.036731 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:46:05.036979 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:05.036984 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.037877 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.037879 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:05.038801 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:05.038805 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.039418 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.039419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:05.039574 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:05.040115 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:46:05.040125 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:46:05.041778 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:46:05.043660 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:46:05.043667 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:46:05.043670 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:46:05.044014 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:46:05.044277 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:46:05.044278 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:46:05.044278 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:46:05.044438 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:46:05.044821 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:46:05.044828 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:46:05.045150 (+ 322us) client_negotiation.cc:769] Sending connection context
0928 04:46:05.045311 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 04:46:05.045512 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for kudu/127.1.226.194@KRBTEST.COM; defaulting to no policy
I20260928 04:46:05.046561 17478 heartbeater.cc:347] Connected to a master server at 127.1.226.254:44913
I20260928 04:46:05.046653 17478 heartbeater.cc:464] Registering TS with master...
I20260928 04:46:05.046864 17478 heartbeater.cc:511] Master 127.1.226.254:44913 requested a full tablet report, sending...
I20260928 04:46:05.047570 17287 ts_manager.cc:194] Registered new tserver with Master: 9c8dc7ae435f4dc4a223d2996000aeda (127.1.226.193:36515)
I20260928 04:46:05.048388 17287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.193@KRBTEST.COM'} at 127.1.226.193:36743
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
I20260928 04:46:05.063397 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:0
--local_ip_for_outbound_sockets=127.1.226.194
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:44913
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:46:05.147775 17483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:46:05.148032 17483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:46:05.148061 17483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:46:05.149362 17483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:46:05.149435 17483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:46:05.149466 17483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:46:05.150905 17483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:44913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:46:05.151094 17483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:46:05.151286 17483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:46:05.156508 17491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:46:05.156543 17488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:46:05.156590 17489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:46:05.156750 17483 server_base.cc:1034] running on GCE node
I20260928 04:46:05.156955 17483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:46:05.157205 17483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:46:05.158368 17483 hybrid_clock.cc:648] HybridClock initialized: now 1790570765158351 us; error 39 us; skew 500 ppm
Sep 28 04:46:05 dist-test-slave-j5c3 krb5kdc[17242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570765, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:46:05.161021 17483 init.cc:377] Logged in from keytab as kudu/127.1.226.194@KRBTEST.COM (short username kudu)
I20260928 04:46:05.161497 17483 webserver.cc:460] Webserver started at http://127.1.226.194:45573/ using document root <none> and password file <none>
I20260928 04:46:05.161729 17483 fs_manager.cc:359] Metadata directory not provided
I20260928 04:46:05.161799 17483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:46:05.161959 17483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:46:05.163040 17483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/data/instance:
uuid: "70efcba756114c0a9d42decf0be343e5"
format_stamp: "Formatted at 2026-09-28 04:46:05 on dist-test-slave-j5c3"
server_key: "20ef71b86b9269b3908bb4e546a98aea"
server_key_iv: "edc59b9ba38ed4e4fdc2a4f9a604cc57"
server_key_version: "encryptionkey@0"
I20260928 04:46:05.163425 17483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance:
uuid: "70efcba756114c0a9d42decf0be343e5"
format_stamp: "Formatted at 2026-09-28 04:46:05 on dist-test-slave-j5c3"
server_key: "20ef71b86b9269b3908bb4e546a98aea"
server_key_iv: "edc59b9ba38ed4e4fdc2a4f9a604cc57"
server_key_version: "encryptionkey@0"
I20260928 04:46:05.164949 17483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:46:05.165766 17498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:46:05.165972 17483 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:46:05.166079 17483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "70efcba756114c0a9d42decf0be343e5"
format_stamp: "Formatted at 2026-09-28 04:46:05 on dist-test-slave-j5c3"
server_key: "20ef71b86b9269b3908bb4e546a98aea"
server_key_iv: "edc59b9ba38ed4e4fdc2a4f9a604cc57"
server_key_version: "encryptionkey@0"
I20260928 04:46:05.166154 17483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:46:05.178488 17483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:46:05.180508 17483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:46:05.180673 17483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:46:05.180917 17483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:46:05.181245 17483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:46:05.181301 17483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:46:05.181342 17483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:46:05.181372 17483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:46:05.187441 17483 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:35391
I20260928 04:46:05.187496 17611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:35391 every 8 connection(s)
I20260928 04:46:05.187867 17483 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
I20260928 04:46:05.188491 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 17483
I20260928 04:46:05.188617 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance
I20260928 04:46:05.188838 1931 external_mini_cluster.cc:1428] Setting key 0ac55b9241b84399baa19ecf6c83a0c0
Sep 28 04:46:05 dist-test-slave-j5c3 krb5kdc[17242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570765, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.226.195@KRBTEST.COM; defaulting to no policy
I20260928 04:46:05.199204 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:05.188700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:44713 (local address 127.1.226.254:44913)
0928 04:46:05.188806 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 04:46:05.188808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:46:05.189258 (+ 450us) server_negotiation.cc:407] Connection header received
0928 04:46:05.189826 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:46:05.189827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:46:05.189841 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:46:05.189908 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:46:05.190831 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.191470 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.193103 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.193369 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.193459 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:05.196224 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:46:05.196227 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:46:05.196227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:46:05.196234 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:46:05.197731 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:46:05.198286 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:46:05.198287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:46:05.198288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:46:05.198342 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:46:05.198580 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:46:05.198580 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:46:05.198580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:46:05.198687 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:46:05.198736 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 04:46:05.199059 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 04:46:05.199132 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 04:46:05.199391 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:05.188657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.194:44713)
0928 04:46:05.189058 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:05.189122 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 04:46:05.189727 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:46:05.189962 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:46:05.189966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:46:05.190537 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:46:05.190732 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:05.190737 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.191573 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.191575 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:05.192988 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:05.192991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.193488 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.193490 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:05.193666 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:05.194230 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:46:05.194242 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:46:05.196079 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:46:05.197829 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:46:05.197839 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:46:05.197842 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:46:05.198193 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:46:05.198411 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:46:05.198411 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:46:05.198412 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:46:05.198523 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:46:05.198742 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:46:05.198746 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:46:05.198945 (+ 199us) client_negotiation.cc:769] Sending connection context
0928 04:46:05.199037 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 04:46:05.199174 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":107,"threads_started":1}
I20260928 04:46:05.199903 17612 heartbeater.cc:347] Connected to a master server at 127.1.226.254:44913
I20260928 04:46:05.200018 17612 heartbeater.cc:464] Registering TS with master...
I20260928 04:46:05.200268 17612 heartbeater.cc:511] Master 127.1.226.254:44913 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.195@KRBTEST.COM" created.
I20260928 04:46:05.200762 17287 ts_manager.cc:194] Registered new tserver with Master: 70efcba756114c0a9d42decf0be343e5 (127.1.226.194:35391)
I20260928 04:46:05.201151 17287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.194@KRBTEST.COM'} at 127.1.226.194:44713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
I20260928 04:46:05.216738 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:0
--local_ip_for_outbound_sockets=127.1.226.195
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:44913
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:46:05.302279 17617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:46:05.302507 17617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:46:05.302533 17617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:46:05.303670 17617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:46:05.303710 17617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:46:05.303736 17617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:46:05.305225 17617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:44913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:46:05.305433 17617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:46:05.305630 17617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:46:05.310817 17625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:46:05.310822 17622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:46:05.311025 17623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:46:05.311045 17617 server_base.cc:1034] running on GCE node
I20260928 04:46:05.311369 17617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:46:05.311633 17617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:46:05.312824 17617 hybrid_clock.cc:648] HybridClock initialized: now 1790570765312805 us; error 43 us; skew 500 ppm
Sep 28 04:46:05 dist-test-slave-j5c3 krb5kdc[17242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570765, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:46:05.315385 17617 init.cc:377] Logged in from keytab as kudu/127.1.226.195@KRBTEST.COM (short username kudu)
I20260928 04:46:05.315840 17617 webserver.cc:460] Webserver started at http://127.1.226.195:35757/ using document root <none> and password file <none>
I20260928 04:46:05.316087 17617 fs_manager.cc:359] Metadata directory not provided
I20260928 04:46:05.316156 17617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:46:05.316332 17617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:46:05.317413 17617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/data/instance:
uuid: "b9b5acc492f8432199d9036830588cec"
format_stamp: "Formatted at 2026-09-28 04:46:05 on dist-test-slave-j5c3"
server_key: "796f9d6088dd4bc5b5eda85f4b6168c4"
server_key_iv: "7950e6942295306e6dc8c2df1a45a47e"
server_key_version: "encryptionkey@0"
I20260928 04:46:05.317858 17617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance:
uuid: "b9b5acc492f8432199d9036830588cec"
format_stamp: "Formatted at 2026-09-28 04:46:05 on dist-test-slave-j5c3"
server_key: "796f9d6088dd4bc5b5eda85f4b6168c4"
server_key_iv: "7950e6942295306e6dc8c2df1a45a47e"
server_key_version: "encryptionkey@0"
I20260928 04:46:05.319311 17617 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:46:05.320070 17632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:46:05.320257 17617 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:46:05.320389 17617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "b9b5acc492f8432199d9036830588cec"
format_stamp: "Formatted at 2026-09-28 04:46:05 on dist-test-slave-j5c3"
server_key: "796f9d6088dd4bc5b5eda85f4b6168c4"
server_key_iv: "7950e6942295306e6dc8c2df1a45a47e"
server_key_version: "encryptionkey@0"
I20260928 04:46:05.320473 17617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:46:05.332486 17617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:46:05.334403 17617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:46:05.334571 17617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:46:05.334807 17617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:46:05.335144 17617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:46:05.335201 17617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:46:05.335237 17617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:46:05.335263 17617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:46:05.341514 17617 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:43639
I20260928 04:46:05.341590 17745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:43639 every 8 connection(s)
I20260928 04:46:05.341946 17617 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
I20260928 04:46:05.342309 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 17617
I20260928 04:46:05.342423 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance
I20260928 04:46:05.342608 1931 external_mini_cluster.cc:1428] Setting key 5345b74aa2f761ef9fc78275614b42ee
Sep 28 04:46:05 dist-test-slave-j5c3 krb5kdc[17242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570765, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:46:05.351673 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:05.343082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:60215 (local address 127.1.226.254:44913)
0928 04:46:05.343128 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 04:46:05.343130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:46:05.343139 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:46:05.343897 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:46:05.343898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:46:05.343908 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:46:05.343954 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:46:05.345012 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.345542 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.346558 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.346752 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.346816 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:05.349092 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:46:05.349093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:46:05.349094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:46:05.349100 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:46:05.350344 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:46:05.350724 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:46:05.350725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:46:05.350725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:46:05.350816 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:46:05.351109 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:46:05.351109 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:46:05.351110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:46:05.351217 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:46:05.351269 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 04:46:05.351536 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 04:46:05.351603 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:46:05.351909 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:05.342794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.195:60215)
0928 04:46:05.342966 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:05.343017 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 04:46:05.343740 (+ 723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:46:05.344030 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:46:05.344033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:46:05.344706 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:46:05.344918 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:05.344924 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.345632 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.345633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:05.346472 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:05.346474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.346840 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.346841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:05.346960 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:05.347565 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:46:05.347574 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:46:05.349008 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:46:05.350424 (+ 1416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:46:05.350430 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:46:05.350432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:46:05.350658 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:46:05.350917 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:46:05.350918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:46:05.350918 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:46:05.351043 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:46:05.351277 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:46:05.351282 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:46:05.351417 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 04:46:05.351509 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 04:46:05.351648 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260928 04:46:05.352541 17746 heartbeater.cc:347] Connected to a master server at 127.1.226.254:44913
I20260928 04:46:05.352634 17746 heartbeater.cc:464] Registering TS with master...
I20260928 04:46:05.352809 17746 heartbeater.cc:511] Master 127.1.226.254:44913 requested a full tablet report, sending...
I20260928 04:46:05.353227 17287 ts_manager.cc:194] Registered new tserver with Master: b9b5acc492f8432199d9036830588cec (127.1.226.195:43639)
I20260928 04:46:05.353567 17287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.195@KRBTEST.COM'} at 127.1.226.195:60215
I20260928 04:46:05.354521 1931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:46:05.362335 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:05.355966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37124 (local address 127.1.226.254:44913)
0928 04:46:05.356028 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 04:46:05.356030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:46:05.356093 (+ 63us) server_negotiation.cc:407] Connection header received
0928 04:46:05.356184 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:46:05.356185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:46:05.356195 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:46:05.356251 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:46:05.356818 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.357309 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.358061 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.358253 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.358331 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:05.359674 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:46:05.359676 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:46:05.359677 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:46:05.359684 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:46:05.361128 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:46:05.361497 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:46:05.361499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:46:05.361499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:46:05.361559 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:46:05.361766 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:46:05.361766 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:46:05.361767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:46:05.361916 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:46:05.362019 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 04:46:05.362152 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 04:46:05.362232 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 04:46:05.364009 17287 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:46:05.365646 17287 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:46:05.372378 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:05.367863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:35391 (local address 127.0.0.1:32802)
0928 04:46:05.367996 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:05.368036 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 04:46:05.368096 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:46:05.368435 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:46:05.368438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:46:05.368440 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:46:05.368505 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:05.368508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.369729 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.369730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:05.370943 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:05.370946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.372005 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.372007 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:05.372082 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:05.372095 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 04:46:05.372237 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 04:46:05.372298 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260928 04:46:05.372682 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:05.368284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:43639 (local address 127.0.0.1:49104)
0928 04:46:05.368408 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:05.368475 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 04:46:05.368530 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:46:05.368922 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:46:05.368923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:46:05.368926 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:46:05.368984 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:05.368986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.370322 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.370323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:05.371386 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:05.371388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.372202 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.372203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:05.372262 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:05.372266 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:46:05.372327 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 04:46:05.372367 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":32,"thread_start_us":55,"threads_started":1}
I20260928 04:46:05.372869 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:05.368086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:36515 (local address 127.0.0.1:38420)
0928 04:46:05.368293 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:05.368304 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:46:05.368349 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:46:05.369012 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:46:05.369013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:46:05.369015 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:46:05.369057 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:05.369059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.370349 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.370350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:05.371336 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:05.371339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.372312 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.372313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:05.372371 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:05.372375 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:46:05.372443 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 04:46:05.372576 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":50,"threads_started":1}
I20260928 04:46:05.372866 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:05.368334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38420 (local address 127.1.226.193:36515)
0928 04:46:05.368794 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0928 04:46:05.368798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:46:05.368807 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:46:05.368844 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:46:05.368846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:46:05.368861 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:46:05.368992 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:46:05.369124 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.370264 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.371447 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.372143 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.372225 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:05.372280 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 04:46:05.372672 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 04:46:05.372775 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":81,"threads_started":1}
I20260928 04:46:05.372962 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:05.367979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32802 (local address 127.1.226.194:35391)
0928 04:46:05.368176 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 04:46:05.368182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:46:05.368194 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:46:05.368237 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:46:05.368239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:46:05.368256 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:46:05.368372 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:46:05.368590 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.369598 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.371038 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.371836 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.371913 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:05.371970 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 04:46:05.372511 (+ 541us) server_negotiation.cc:299] Negotiation successful
0928 04:46:05.372560 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
I20260928 04:46:05.373416 17759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:05.368584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49104 (local address 127.1.226.195:43639)
0928 04:46:05.368738 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 04:46:05.368742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:46:05.368754 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:46:05.368795 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:46:05.368797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:46:05.368815 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:46:05.369060 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:46:05.369127 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.370203 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.371470 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.372090 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.372196 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:05.372252 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 04:46:05.372341 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 04:46:05.373337 (+ 996us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260928 04:46:05.373613 17412 tablet_service.cc:1467] Processing CreateTablet for tablet 6ee59c6c89614680885835bf871e8bf7 (DEFAULT_TABLE table=test-table [id=d65dadd535f44c2a8659022cc903595b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:46:05.373648 17546 tablet_service.cc:1467] Processing CreateTablet for tablet 6ee59c6c89614680885835bf871e8bf7 (DEFAULT_TABLE table=test-table [id=d65dadd535f44c2a8659022cc903595b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:46:05.374001 17412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ee59c6c89614680885835bf871e8bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:46:05.374001 17546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ee59c6c89614680885835bf871e8bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:46:05.374141 17680 tablet_service.cc:1467] Processing CreateTablet for tablet 6ee59c6c89614680885835bf871e8bf7 (DEFAULT_TABLE table=test-table [id=d65dadd535f44c2a8659022cc903595b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:46:05.374430 17680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ee59c6c89614680885835bf871e8bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:46:05.376911 17760 tablet_bootstrap.cc:492] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda: Bootstrap starting.
I20260928 04:46:05.377506 17761 tablet_bootstrap.cc:492] T 6ee59c6c89614680885835bf871e8bf7 P 70efcba756114c0a9d42decf0be343e5: Bootstrap starting.
I20260928 04:46:05.377600 17760 tablet_bootstrap.cc:654] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda: Neither blocks nor log segments found. Creating new log.
I20260928 04:46:05.377676 17762 tablet_bootstrap.cc:492] T 6ee59c6c89614680885835bf871e8bf7 P b9b5acc492f8432199d9036830588cec: Bootstrap starting.
I20260928 04:46:05.378170 17762 tablet_bootstrap.cc:654] T 6ee59c6c89614680885835bf871e8bf7 P b9b5acc492f8432199d9036830588cec: Neither blocks nor log segments found. Creating new log.
I20260928 04:46:05.378196 17760 log.cc:824] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda: Log is configured to *not* fsync() on all Append() calls
I20260928 04:46:05.378445 17761 tablet_bootstrap.cc:654] T 6ee59c6c89614680885835bf871e8bf7 P 70efcba756114c0a9d42decf0be343e5: Neither blocks nor log segments found. Creating new log.
I20260928 04:46:05.378723 17762 log.cc:824] T 6ee59c6c89614680885835bf871e8bf7 P b9b5acc492f8432199d9036830588cec: Log is configured to *not* fsync() on all Append() calls
I20260928 04:46:05.379031 17761 log.cc:824] T 6ee59c6c89614680885835bf871e8bf7 P 70efcba756114c0a9d42decf0be343e5: Log is configured to *not* fsync() on all Append() calls
I20260928 04:46:05.379240 17760 tablet_bootstrap.cc:492] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda: No bootstrap required, opened a new log
I20260928 04:46:05.379318 17760 ts_tablet_manager.cc:1397] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:46:05.379601 17762 tablet_bootstrap.cc:492] T 6ee59c6c89614680885835bf871e8bf7 P b9b5acc492f8432199d9036830588cec: No bootstrap required, opened a new log
I20260928 04:46:05.379671 17762 ts_tablet_manager.cc:1397] T 6ee59c6c89614680885835bf871e8bf7 P b9b5acc492f8432199d9036830588cec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:46:05.379724 17761 tablet_bootstrap.cc:492] T 6ee59c6c89614680885835bf871e8bf7 P 70efcba756114c0a9d42decf0be343e5: No bootstrap required, opened a new log
I20260928 04:46:05.379799 17761 ts_tablet_manager.cc:1397] T 6ee59c6c89614680885835bf871e8bf7 P 70efcba756114c0a9d42decf0be343e5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:46:05.381179 17761 raft_consensus.cc:348] T 6ee59c6c89614680885835bf871e8bf7 P 70efcba756114c0a9d42decf0be343e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } }
I20260928 04:46:05.381348 17761 raft_consensus.cc:374] T 6ee59c6c89614680885835bf871e8bf7 P 70efcba756114c0a9d42decf0be343e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:46:05.381392 17761 raft_consensus.cc:729] T 6ee59c6c89614680885835bf871e8bf7 P 70efcba756114c0a9d42decf0be343e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70efcba756114c0a9d42decf0be343e5, State: Initialized, Role: FOLLOWER
I20260928 04:46:05.381361 17760 raft_consensus.cc:348] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } }
I20260928 04:46:05.381553 17760 raft_consensus.cc:374] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:46:05.381520 17761 consensus_queue.cc:260] T 6ee59c6c89614680885835bf871e8bf7 P 70efcba756114c0a9d42decf0be343e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } }
I20260928 04:46:05.381600 17760 raft_consensus.cc:729] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c8dc7ae435f4dc4a223d2996000aeda, State: Initialized, Role: FOLLOWER
I20260928 04:46:05.381599 17762 raft_consensus.cc:348] T 6ee59c6c89614680885835bf871e8bf7 P b9b5acc492f8432199d9036830588cec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } }
I20260928 04:46:05.381752 17762 raft_consensus.cc:374] T 6ee59c6c89614680885835bf871e8bf7 P b9b5acc492f8432199d9036830588cec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:46:05.381791 17762 raft_consensus.cc:729] T 6ee59c6c89614680885835bf871e8bf7 P b9b5acc492f8432199d9036830588cec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9b5acc492f8432199d9036830588cec, State: Initialized, Role: FOLLOWER
I20260928 04:46:05.381749 17760 consensus_queue.cc:260] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } }
I20260928 04:46:05.381822 17761 ts_tablet_manager.cc:1428] T 6ee59c6c89614680885835bf871e8bf7 P 70efcba756114c0a9d42decf0be343e5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:46:05.381939 17612 heartbeater.cc:503] Master 127.1.226.254:44913 was elected leader, sending a full tablet report...
I20260928 04:46:05.381933 17762 consensus_queue.cc:260] T 6ee59c6c89614680885835bf871e8bf7 P b9b5acc492f8432199d9036830588cec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } }
I20260928 04:46:05.382015 17760 ts_tablet_manager.cc:1428] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:46:05.382154 17478 heartbeater.cc:503] Master 127.1.226.254:44913 was elected leader, sending a full tablet report...
I20260928 04:46:05.382200 17762 ts_tablet_manager.cc:1428] T 6ee59c6c89614680885835bf871e8bf7 P b9b5acc492f8432199d9036830588cec: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 04:46:05.382308 17746 heartbeater.cc:503] Master 127.1.226.254:44913 was elected leader, sending a full tablet report...
W20260928 04:46:05.438508 17613 tablet.cc:2369] T 6ee59c6c89614680885835bf871e8bf7 P 70efcba756114c0a9d42decf0be343e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:46:05.498375 17767 raft_consensus.cc:482] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:46:05.498565 17767 raft_consensus.cc:504] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } }
I20260928 04:46:05.498965 17767 leader_election.cc:288] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70efcba756114c0a9d42decf0be343e5 (127.1.226.194:35391), b9b5acc492f8432199d9036830588cec (127.1.226.195:43639)
I20260928 04:46:05.501948 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:05.499192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:35391 (local address 127.1.226.193:59687)
0928 04:46:05.499252 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:05.499266 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:46:05.499314 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:46:05.499592 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:46:05.499594 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:46:05.499596 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:46:05.499669 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:05.499671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.500341 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.500342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:05.501086 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:05.501088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.501667 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.501668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:05.501756 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:05.501759 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:46:05.501828 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 04:46:05.501873 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 04:46:05.501948 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:05.499321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:59687 (local address 127.1.226.194:35391)
0928 04:46:05.499445 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 04:46:05.499448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:46:05.499457 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:46:05.499489 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:46:05.499490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:46:05.499499 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:46:05.499584 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:46:05.499745 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.500249 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.501198 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.501581 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.501669 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:05.501679 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:46:05.501834 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 04:46:05.501873 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260928 04:46:05.502206 17759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:05.499424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:40765 (local address 127.1.226.195:43639)
0928 04:46:05.499471 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 04:46:05.499473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:46:05.499615 (+ 142us) server_negotiation.cc:407] Connection header received
0928 04:46:05.499688 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:46:05.499690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:46:05.499700 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:46:05.499756 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:46:05.499963 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.500495 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.501431 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.501810 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.501893 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:05.501901 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 04:46:05.502088 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 04:46:05.502141 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:46:05.502348 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee59c6c89614680885835bf871e8bf7" candidate_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70efcba756114c0a9d42decf0be343e5" is_pre_election: true
I20260928 04:46:05.502439 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:05.499344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:43639 (local address 127.1.226.193:40765)
0928 04:46:05.499529 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:05.499538 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:46:05.499629 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:46:05.499815 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:46:05.499815 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:46:05.499817 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:46:05.499885 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:05.499886 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.500578 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.500579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:05.501333 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:05.501335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.501894 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.501895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:05.501967 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:05.501971 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:46:05.502320 (+ 349us) client_negotiation.cc:240] Negotiation successful
0928 04:46:05.502362 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":127,"threads_started":1}
I20260928 04:46:05.502535 17566 raft_consensus.cc:2454] T 6ee59c6c89614680885835bf871e8bf7 P 70efcba756114c0a9d42decf0be343e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c8dc7ae435f4dc4a223d2996000aeda in term 0.
I20260928 04:46:05.502799 17367 leader_election.cc:302] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70efcba756114c0a9d42decf0be343e5, 9c8dc7ae435f4dc4a223d2996000aeda; no voters:
I20260928 04:46:05.502799 17700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee59c6c89614680885835bf871e8bf7" candidate_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9b5acc492f8432199d9036830588cec" is_pre_election: true
I20260928 04:46:05.502938 17767 raft_consensus.cc:2781] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:46:05.502956 17700 raft_consensus.cc:2454] T 6ee59c6c89614680885835bf871e8bf7 P b9b5acc492f8432199d9036830588cec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c8dc7ae435f4dc4a223d2996000aeda in term 0.
I20260928 04:46:05.502995 17767 raft_consensus.cc:482] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:46:05.503012 17767 raft_consensus.cc:3037] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:46:05.503773 17767 raft_consensus.cc:504] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } }
I20260928 04:46:05.503903 17767 leader_election.cc:288] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda [CANDIDATE]: Term 1 election: Requested vote from peers 70efcba756114c0a9d42decf0be343e5 (127.1.226.194:35391), b9b5acc492f8432199d9036830588cec (127.1.226.195:43639)
I20260928 04:46:05.504101 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee59c6c89614680885835bf871e8bf7" candidate_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70efcba756114c0a9d42decf0be343e5"
I20260928 04:46:05.504101 17700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee59c6c89614680885835bf871e8bf7" candidate_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9b5acc492f8432199d9036830588cec"
I20260928 04:46:05.504206 17566 raft_consensus.cc:3037] T 6ee59c6c89614680885835bf871e8bf7 P 70efcba756114c0a9d42decf0be343e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:46:05.504206 17700 raft_consensus.cc:3037] T 6ee59c6c89614680885835bf871e8bf7 P b9b5acc492f8432199d9036830588cec [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:46:05.505120 17700 raft_consensus.cc:2454] T 6ee59c6c89614680885835bf871e8bf7 P b9b5acc492f8432199d9036830588cec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c8dc7ae435f4dc4a223d2996000aeda in term 1.
I20260928 04:46:05.505120 17566 raft_consensus.cc:2454] T 6ee59c6c89614680885835bf871e8bf7 P 70efcba756114c0a9d42decf0be343e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c8dc7ae435f4dc4a223d2996000aeda in term 1.
I20260928 04:46:05.505317 17367 leader_election.cc:302] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c8dc7ae435f4dc4a223d2996000aeda, b9b5acc492f8432199d9036830588cec; no voters:
I20260928 04:46:05.505407 17767 raft_consensus.cc:2781] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:46:05.505564 17767 raft_consensus.cc:686] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda [term 1 LEADER]: Becoming Leader. State: Replica: 9c8dc7ae435f4dc4a223d2996000aeda, State: Running, Role: LEADER
I20260928 04:46:05.505661 17767 consensus_queue.cc:237] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } }
I20260928 04:46:05.506533 17286 catalog_manager.cc:5495] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c8dc7ae435f4dc4a223d2996000aeda (127.1.226.193). New cstate: current_term: 1 leader_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } health_report { overall_health: UNKNOWN } } }
I20260928 04:46:05.516017 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:05.513632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38428 (local address 127.1.226.193:36515)
0928 04:46:05.513673 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 04:46:05.513675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:46:05.513684 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:46:05.513740 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:46:05.513741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:46:05.513757 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:46:05.513879 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:46:05.513987 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.514452 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.515224 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:05.515427 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:05.515503 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:05.515675 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:46:05.515791 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:46:05.515892 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 04:46:05.515911 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 04:46:05.515932 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 04:46:05.517913 17700 raft_consensus.cc:1261] T 6ee59c6c89614680885835bf871e8bf7 P b9b5acc492f8432199d9036830588cec [term 1 FOLLOWER]: Refusing update from remote peer 9c8dc7ae435f4dc4a223d2996000aeda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:46:05.517913 17566 raft_consensus.cc:1261] T 6ee59c6c89614680885835bf871e8bf7 P 70efcba756114c0a9d42decf0be343e5 [term 1 FOLLOWER]: Refusing update from remote peer 9c8dc7ae435f4dc4a223d2996000aeda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:46:05.518317 17770 consensus_queue.cc:1035] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda [LEADER]: Connected to new peer: Peer: permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:46:05.518455 17767 consensus_queue.cc:1035] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:46:05.523589 17775 mvcc.cc:205] Tried to move back new op lower bound from 7334177855558176768 to 7334177855511826432. Current Snapshot: MvccSnapshot[applied={T|T < 7334177855558176768 or (T in {7334177855558176768})}]
I20260928 04:46:05.523965 17286 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:46:05.524087 17286 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:46:05.524995 17286 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504: Sending DeleteTablet for 3 replicas of tablet 6ee59c6c89614680885835bf871e8bf7
I20260928 04:46:05.525411 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 17349
I20260928 04:46:05.525403 17546 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ee59c6c89614680885835bf871e8bf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:46:05 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:32802
I20260928 04:46:05.525568 17680 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ee59c6c89614680885835bf871e8bf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:46:05 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:49104
I20260928 04:46:05.526188 17782 tablet_replica.cc:330] T 6ee59c6c89614680885835bf871e8bf7 P 70efcba756114c0a9d42decf0be343e5: stopping tablet replica
I20260928 04:46:05.526300 17782 raft_consensus.cc:2229] T 6ee59c6c89614680885835bf871e8bf7 P 70efcba756114c0a9d42decf0be343e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:46:05.526418 17782 raft_consensus.cc:2258] T 6ee59c6c89614680885835bf871e8bf7 P 70efcba756114c0a9d42decf0be343e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:46:05.526918 17782 ts_tablet_manager.cc:1905] T 6ee59c6c89614680885835bf871e8bf7 P 70efcba756114c0a9d42decf0be343e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:46:05.529115 17782 ts_tablet_manager.cc:1918] T 6ee59c6c89614680885835bf871e8bf7 P 70efcba756114c0a9d42decf0be343e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:46:05.529188 17782 log.cc:1196] T 6ee59c6c89614680885835bf871e8bf7 P 70efcba756114c0a9d42decf0be343e5: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/wal/wals/6ee59c6c89614680885835bf871e8bf7
I20260928 04:46:05.529546 17782 ts_tablet_manager.cc:1939] T 6ee59c6c89614680885835bf871e8bf7 P 70efcba756114c0a9d42decf0be343e5: Deleting consensus metadata
I20260928 04:46:05.530066 17783 tablet_replica.cc:330] T 6ee59c6c89614680885835bf871e8bf7 P b9b5acc492f8432199d9036830588cec: stopping tablet replica
I20260928 04:46:05.530110 17273 catalog_manager.cc:4841] TS 70efcba756114c0a9d42decf0be343e5 (127.1.226.194:35391): tablet 6ee59c6c89614680885835bf871e8bf7 (table test-table [id=d65dadd535f44c2a8659022cc903595b]) successfully deleted
I20260928 04:46:05.530207 17783 raft_consensus.cc:2229] T 6ee59c6c89614680885835bf871e8bf7 P b9b5acc492f8432199d9036830588cec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:46:05.530367 17783 raft_consensus.cc:2258] T 6ee59c6c89614680885835bf871e8bf7 P b9b5acc492f8432199d9036830588cec [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 04:46:05.530972 17273 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.226.193:36515 (error 108)
I20260928 04:46:05.531014 17783 ts_tablet_manager.cc:1905] T 6ee59c6c89614680885835bf871e8bf7 P b9b5acc492f8432199d9036830588cec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:46:05.531201 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 17483
I20260928 04:46:05.533506 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:05.533290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:36515 (local address 127.0.0.1:38440)
0928 04:46:05.533372 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:05.533466 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.193:36515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 04:46:05.533663 17273 catalog_manager.cc:4584] TS 9c8dc7ae435f4dc4a223d2996000aeda (127.1.226.193:36515): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6ee59c6c89614680885835bf871e8bf7: Network error: Client connection negotiation failed: client connection to 127.1.226.193:36515: connect: Connection refused (error 111)
I20260928 04:46:05.535135 17783 ts_tablet_manager.cc:1918] T 6ee59c6c89614680885835bf871e8bf7 P b9b5acc492f8432199d9036830588cec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:46:05.535230 17783 log.cc:1196] T 6ee59c6c89614680885835bf871e8bf7 P b9b5acc492f8432199d9036830588cec: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/wal/wals/6ee59c6c89614680885835bf871e8bf7
I20260928 04:46:05.535537 17783 ts_tablet_manager.cc:1939] T 6ee59c6c89614680885835bf871e8bf7 P b9b5acc492f8432199d9036830588cec: Deleting consensus metadata
I20260928 04:46:05.536046 17273 catalog_manager.cc:4841] TS b9b5acc492f8432199d9036830588cec (127.1.226.195:43639): tablet 6ee59c6c89614680885835bf871e8bf7 (table test-table [id=d65dadd535f44c2a8659022cc903595b]) successfully deleted
I20260928 04:46:05.537813 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 17617
I20260928 04:46:05.542899 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 17255
I20260928 04:46:25.549009 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:44913
--webserver_interface=127.1.226.254
--webserver_port=45277
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:44913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:46:25.632548 17785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:46:25.632771 17785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:46:25.632793 17785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:46:25.633898 17785 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 04:46:25.633932 17785 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 04:46:25.633946 17785 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 04:46:25.633957 17785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:46:25.633970 17785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:46:25.633983 17785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:46:25.633996 17785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:46:25.634007 17785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:46:25.635537 17785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:44913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:44913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=45277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:46:25.635748 17785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:46:25.635927 17785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:46:25.644711 17791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:46:25.644812 17792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:46:25.644908 17785 server_base.cc:1034] running on GCE node
W20260928 04:46:25.644714 17794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:46:25.645256 17785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:46:25.645597 17785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:46:25.646963 17785 hybrid_clock.cc:648] HybridClock initialized: now 1790570785646959 us; error 37 us; skew 500 ppm
Sep 28 04:46:25 dist-test-slave-j5c3 krb5kdc[17242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570785, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:46:25.649765 17785 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:46:25.650307 17785 webserver.cc:460] Webserver started at http://127.1.226.254:45277/ using document root <none> and password file <none>
I20260928 04:46:25.650550 17785 fs_manager.cc:359] Metadata directory not provided
I20260928 04:46:25.650632 17785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:46:25.652077 17785 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:46:25.652789 17801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:46:25.652959 17785 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:46:25.653069 17785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "7de1202fbe4b4f80bdb46a2663c75504"
format_stamp: "Formatted at 2026-09-28 04:46:04 on dist-test-slave-j5c3"
server_key: "cb44743a5d0eed75b4d08779cfd9454a"
server_key_iv: "b4cd6b9ca4fceb692159fcaac1aa626f"
server_key_version: "encryptionkey@0"
I20260928 04:46:25.653483 17785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:46:25.664737 17785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:46:25.666707 17785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:46:25.666888 17785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:46:25.671288 17785 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:44913
I20260928 04:46:25.671341 17853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:44913 every 8 connection(s)
I20260928 04:46:25.671756 17785 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:46:25.673369 17854 sys_catalog.cc:263] Verifying existing consensus state
I20260928 04:46:25.674113 17854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504: Bootstrap starting.
I20260928 04:46:25.674777 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 17785
I20260928 04:46:25.675082 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:36515
--local_ip_for_outbound_sockets=127.1.226.193
--tserver_master_addrs=127.1.226.254:44913
--webserver_port=32933
--webserver_interface=127.1.226.193
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:46:25.683645 17854 log.cc:824] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504: Log is configured to *not* fsync() on all Append() calls
I20260928 04:46:25.685760 17854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:46:25.686167 17854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504: Bootstrap complete.
I20260928 04:46:25.688596 17854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de1202fbe4b4f80bdb46a2663c75504" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44913 } }
I20260928 04:46:25.688834 17854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7de1202fbe4b4f80bdb46a2663c75504, State: Initialized, Role: FOLLOWER
I20260928 04:46:25.689026 17854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de1202fbe4b4f80bdb46a2663c75504" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44913 } }
I20260928 04:46:25.689088 17854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:46:25.689158 17854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:46:25.689209 17854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [term 1 FOLLOWER]: Advancing to term 2
I20260928 04:46:25.690364 17854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de1202fbe4b4f80bdb46a2663c75504" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44913 } }
I20260928 04:46:25.690457 17854 leader_election.cc:302] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7de1202fbe4b4f80bdb46a2663c75504; no voters:
I20260928 04:46:25.690651 17854 leader_election.cc:288] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 04:46:25.690699 17858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 04:46:25.690881 17858 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [term 2 LEADER]: Becoming Leader. State: Replica: 7de1202fbe4b4f80bdb46a2663c75504, State: Running, Role: LEADER
I20260928 04:46:25.690945 17854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:46:25.691027 17858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de1202fbe4b4f80bdb46a2663c75504" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44913 } }
I20260928 04:46:25.691293 17858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7de1202fbe4b4f80bdb46a2663c75504. Latest consensus state: current_term: 2 leader_uuid: "7de1202fbe4b4f80bdb46a2663c75504" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de1202fbe4b4f80bdb46a2663c75504" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44913 } } }
I20260928 04:46:25.691370 17858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [sys.catalog]: This master's current role is: LEADER
I20260928 04:46:25.691319 17859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7de1202fbe4b4f80bdb46a2663c75504" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de1202fbe4b4f80bdb46a2663c75504" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44913 } } }
I20260928 04:46:25.691540 17859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [sys.catalog]: This master's current role is: LEADER
I20260928 04:46:25.692128 17864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:46:25.692946 17864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:46:25.693065 17864 catalog_manager.cc:1259] Loaded cluster ID: fa05fa53151e461ca22e53a29d32bbe1
I20260928 04:46:25.693109 17864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:46:25.694319 17864 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:46:25.694494 17864 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504: Loaded TSK: 0
I20260928 04:46:25.704066 17864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504: Generated new TSK 1
I20260928 04:46:25.704210 17864 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 04:46:25.762895 17856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:46:25.763136 17856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:46:25.763162 17856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:46:25.764488 17856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:46:25.764551 17856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:46:25.764580 17856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:46:25.766196 17856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:36515
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=32933
--tserver_master_addrs=127.1.226.254:44913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:46:25.766476 17856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:46:25.766722 17856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:46:25.772531 17856 server_base.cc:1034] running on GCE node
W20260928 04:46:25.772439 17881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:46:25.772562 17882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:46:25.772436 17884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:46:25.773025 17856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:46:25.773306 17856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:46:25.774464 17856 hybrid_clock.cc:648] HybridClock initialized: now 1790570785774443 us; error 38 us; skew 500 ppm
Sep 28 04:46:25 dist-test-slave-j5c3 krb5kdc[17242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570785, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:46:25.777328 17856 init.cc:377] Logged in from keytab as kudu/127.1.226.193@KRBTEST.COM (short username kudu)
I20260928 04:46:25.777930 17856 webserver.cc:460] Webserver started at http://127.1.226.193:32933/ using document root <none> and password file <none>
I20260928 04:46:25.778208 17856 fs_manager.cc:359] Metadata directory not provided
I20260928 04:46:25.778296 17856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:46:25.780175 17856 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:46:25.781059 17891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:46:25.781268 17856 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:46:25.781366 17856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "9c8dc7ae435f4dc4a223d2996000aeda"
format_stamp: "Formatted at 2026-09-28 04:46:05 on dist-test-slave-j5c3"
server_key: "6555c58716ad430b02df54ea4ec3b1ff"
server_key_iv: "c1e73899d9b3de377b1a75a6f446a72b"
server_key_version: "encryptionkey@0"
I20260928 04:46:25.781812 17856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:46:25.809092 17856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:46:25.811725 17856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:46:25.811906 17856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:46:25.812219 17856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:46:25.812813 17898 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 04:46:25.814355 17856 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 04:46:25.814417 17856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 04:46:25.814450 17856 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 04:46:25.815284 17856 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 04:46:25.815325 17856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 04:46:25.815428 17898 tablet_bootstrap.cc:492] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda: Bootstrap starting.
I20260928 04:46:25.822145 17856 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:36515
I20260928 04:46:25.822470 18005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:36515 every 8 connection(s)
I20260928 04:46:25.822592 17856 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
I20260928 04:46:25.827071 17898 log.cc:824] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda: Log is configured to *not* fsync() on all Append() calls
I20260928 04:46:25.829643 17898 tablet_bootstrap.cc:492] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:46:25.830017 17898 tablet_bootstrap.cc:492] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda: Bootstrap complete.
I20260928 04:46:25.831197 17898 ts_tablet_manager.cc:1397] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.009s
Sep 28 04:46:25 dist-test-slave-j5c3 krb5kdc[17242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570785, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:46:25.831660 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 17856
I20260928 04:46:25.831939 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:35391
--local_ip_for_outbound_sockets=127.1.226.194
--tserver_master_addrs=127.1.226.254:44913
--webserver_port=45573
--webserver_interface=127.1.226.194
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:46:25.832861 17898 raft_consensus.cc:348] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } }
I20260928 04:46:25.833242 17898 raft_consensus.cc:729] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c8dc7ae435f4dc4a223d2996000aeda, State: Initialized, Role: FOLLOWER
I20260928 04:46:25.833419 17898 consensus_queue.cc:260] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } }
I20260928 04:46:25.833702 17898 ts_tablet_manager.cc:1428] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda: Time spent starting tablet: real 0.002s user 0.003s sys 0.002s
I20260928 04:46:25.836239 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:25.823374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.193:46395)
0928 04:46:25.823591 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:25.823640 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 04:46:25.824297 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:46:25.824920 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:46:25.824925 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:46:25.825280 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:46:25.825546 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:25.825552 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:25.827337 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:25.827338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:25.828534 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:25.828537 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:25.829340 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:25.829342 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:25.829494 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:25.830207 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:46:25.830216 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:46:25.832025 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:46:25.834295 (+ 2270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:46:25.834302 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:46:25.834304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:46:25.834633 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:46:25.834934 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:46:25.834935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:46:25.834936 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:46:25.835077 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:46:25.835582 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:46:25.835589 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:46:25.835733 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 04:46:25.835830 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 04:46:25.836005 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":121,"threads_started":1}
I20260928 04:46:25.836746 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:25.823534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:46395 (local address 127.1.226.254:44913)
0928 04:46:25.824322 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0928 04:46:25.824343 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 04:46:25.824424 (+ 81us) server_negotiation.cc:407] Connection header received
0928 04:46:25.824596 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:46:25.824607 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:46:25.824684 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:46:25.824853 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:46:25.825691 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:25.827177 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:25.828625 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:25.829207 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:25.829317 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:25.832140 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:46:25.832144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:46:25.832144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:46:25.832193 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:46:25.834203 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:46:25.834731 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:46:25.834734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:46:25.834734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:46:25.834854 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:46:25.835161 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:46:25.835162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:46:25.835162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:46:25.835472 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:46:25.835552 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 04:46:25.836420 (+ 868us) server_negotiation.cc:299] Negotiation successful
0928 04:46:25.836557 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":81,"threads_started":1}
I20260928 04:46:25.837393 18006 heartbeater.cc:347] Connected to a master server at 127.1.226.254:44913
I20260928 04:46:25.837507 18006 heartbeater.cc:464] Registering TS with master...
I20260928 04:46:25.837738 18006 heartbeater.cc:511] Master 127.1.226.254:44913 requested a full tablet report, sending...
I20260928 04:46:25.838526 17818 ts_manager.cc:194] Registered new tserver with Master: 9c8dc7ae435f4dc4a223d2996000aeda (127.1.226.193:36515)
I20260928 04:46:25.839598 17818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.193@KRBTEST.COM'} at 127.1.226.193:46395
I20260928 04:46:25.840749 18006 heartbeater.cc:503] Master 127.1.226.254:44913 was elected leader, sending a full tablet report...
Sep 28 04:46:25 dist-test-slave-j5c3 krb5kdc[17242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570785, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for kudu/127.1.226.193@KRBTEST.COM
I20260928 04:46:25.847618 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:25.839559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51994 (local address 127.1.226.193:36515)
0928 04:46:25.839722 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 04:46:25.839727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:46:25.839738 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:46:25.839779 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:46:25.839781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:46:25.839799 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:46:25.839995 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:46:25.840514 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:25.841311 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:25.842173 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:25.842473 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:25.842553 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:25.844646 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:46:25.844655 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:46:25.844657 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:46:25.844669 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:46:25.846232 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:46:25.846664 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:46:25.846668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:46:25.846669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:46:25.846733 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:46:25.846946 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:46:25.846947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:46:25.846947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:46:25.847110 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:46:25.847205 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 04:46:25.847406 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 04:46:25.847513 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":76,"threads_started":1}
I20260928 04:46:25.847618 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:25.839254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:36515 (local address 127.0.0.1:51994)
0928 04:46:25.839470 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:25.839508 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 04:46:25.839577 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:46:25.840011 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:46:25.840017 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:46:25.840373 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:46:25.840425 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:25.840428 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:25.841443 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:25.841445 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:25.842070 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:25.842072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:25.842552 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:25.842553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:25.842640 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:25.843156 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:46:25.843161 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:46:25.844541 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:46:25.846344 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:46:25.846353 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:46:25.846355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:46:25.846574 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:46:25.846813 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:46:25.846814 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:46:25.846814 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:46:25.846884 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:46:25.847194 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:46:25.847202 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:46:25.847264 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 04:46:25.847367 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 04:46:25.847513 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":114,"threads_started":1}
I20260928 04:46:25.848387 17940 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ee59c6c89614680885835bf871e8bf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:46:05 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:51994
I20260928 04:46:25.849037 18015 tablet_replica.cc:330] stopping tablet replica
I20260928 04:46:25.849160 18015 raft_consensus.cc:2229] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:46:25.849253 18015 raft_consensus.cc:2258] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:46:25.849596 18015 ts_tablet_manager.cc:1905] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:46:25.851898 18015 ts_tablet_manager.cc:1918] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:46:25.851994 18015 log.cc:1196] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/wal/wals/6ee59c6c89614680885835bf871e8bf7
I20260928 04:46:25.852295 18015 ts_tablet_manager.cc:1939] T 6ee59c6c89614680885835bf871e8bf7 P 9c8dc7ae435f4dc4a223d2996000aeda: Deleting consensus metadata
I20260928 04:46:25.852869 17804 catalog_manager.cc:4841] TS 9c8dc7ae435f4dc4a223d2996000aeda (127.1.226.193:36515): tablet 6ee59c6c89614680885835bf871e8bf7 (table test-table [id=d65dadd535f44c2a8659022cc903595b]) successfully deleted
W20260928 04:46:25.919155 18011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:46:25.919365 18011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:46:25.919389 18011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:46:25.920586 18011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:46:25.920629 18011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:46:25.920655 18011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:46:25.922183 18011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:35391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=45573
--tserver_master_addrs=127.1.226.254:44913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:46:25.922381 18011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:46:25.922616 18011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:46:25.927863 18021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:46:25.927894 18020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:46:25.927856 18023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:46:25.927898 18011 server_base.cc:1034] running on GCE node
I20260928 04:46:25.928450 18011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:46:25.928709 18011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:46:25.929875 18011 hybrid_clock.cc:648] HybridClock initialized: now 1790570785929876 us; error 44 us; skew 500 ppm
Sep 28 04:46:25 dist-test-slave-j5c3 krb5kdc[17242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570785, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:46:25.932488 18011 init.cc:377] Logged in from keytab as kudu/127.1.226.194@KRBTEST.COM (short username kudu)
I20260928 04:46:25.932948 18011 webserver.cc:460] Webserver started at http://127.1.226.194:45573/ using document root <none> and password file <none>
I20260928 04:46:25.933190 18011 fs_manager.cc:359] Metadata directory not provided
I20260928 04:46:25.933247 18011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:46:25.934619 18011 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:46:25.935271 18030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:46:25.935436 18011 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:46:25.935528 18011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "70efcba756114c0a9d42decf0be343e5"
format_stamp: "Formatted at 2026-09-28 04:46:05 on dist-test-slave-j5c3"
server_key: "20ef71b86b9269b3908bb4e546a98aea"
server_key_iv: "edc59b9ba38ed4e4fdc2a4f9a604cc57"
server_key_version: "encryptionkey@0"
I20260928 04:46:25.935940 18011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:46:25.957226 18011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:46:25.959846 18011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:46:25.959966 18011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:46:25.960201 18011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:46:25.960512 18011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:46:25.960543 18011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:46:25.960566 18011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:46:25.960579 18011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:46:25.966325 18011 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:35391
I20260928 04:46:25.966373 18143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:35391 every 8 connection(s)
I20260928 04:46:25.966733 18011 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
I20260928 04:46:25.969079 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 18011
I20260928 04:46:25.969380 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:43639
--local_ip_for_outbound_sockets=127.1.226.195
--tserver_master_addrs=127.1.226.254:44913
--webserver_port=35757
--webserver_interface=127.1.226.195
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 04:46:25 dist-test-slave-j5c3 krb5kdc[17242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570785, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:46:25.976999 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:25.967472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:37753 (local address 127.1.226.254:44913)
0928 04:46:25.967533 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 04:46:25.967535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:46:25.967752 (+ 217us) server_negotiation.cc:407] Connection header received
0928 04:46:25.968291 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:46:25.968292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:46:25.968305 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:46:25.968365 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:46:25.969471 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:25.969983 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:25.971218 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:25.971435 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:25.971506 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:25.974101 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:46:25.974102 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:46:25.974103 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:46:25.974118 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:46:25.975526 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:46:25.975949 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:46:25.975950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:46:25.975950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:46:25.976010 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:46:25.976320 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:46:25.976321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:46:25.976322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:46:25.976444 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:46:25.976526 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 04:46:25.976803 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 04:46:25.976907 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 04:46:25.977365 18146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:25.967439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.194:37753)
0928 04:46:25.967612 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:25.967655 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 04:46:25.968143 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:46:25.968461 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:46:25.968466 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:46:25.969086 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:46:25.969355 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:25.969361 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:25.970086 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:25.970087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:25.971121 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:25.971125 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:25.971535 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:25.971536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:25.971687 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:25.972347 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:46:25.972355 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:46:25.973980 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:46:25.975630 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:46:25.975638 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:46:25.975640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:46:25.975869 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:46:25.976079 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:46:25.976080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:46:25.976080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:46:25.976254 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:46:25.976517 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:46:25.976524 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:46:25.976670 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 04:46:25.976771 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 04:46:25.976933 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":96,"threads_started":1}
I20260928 04:46:25.977973 18144 heartbeater.cc:347] Connected to a master server at 127.1.226.254:44913
I20260928 04:46:25.978086 18144 heartbeater.cc:464] Registering TS with master...
I20260928 04:46:25.978292 18144 heartbeater.cc:511] Master 127.1.226.254:44913 requested a full tablet report, sending...
I20260928 04:46:25.978720 17818 ts_manager.cc:194] Registered new tserver with Master: 70efcba756114c0a9d42decf0be343e5 (127.1.226.194:35391)
I20260928 04:46:25.979089 17818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.194@KRBTEST.COM'} at 127.1.226.194:37753
W20260928 04:46:26.053673 18147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:46:26.053934 18147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:46:26.053958 18147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:46:26.055290 18147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:46:26.055404 18147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:46:26.055452 18147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:46:26.057068 18147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:43639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=35757
--tserver_master_addrs=127.1.226.254:44913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:46:26.057325 18147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:46:26.057562 18147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:46:26.062660 18153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:46:26.062618 18155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:46:26.062645 18147 server_base.cc:1034] running on GCE node
W20260928 04:46:26.062615 18152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:46:26.063069 18147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:46:26.063261 18147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:46:26.064390 18147 hybrid_clock.cc:648] HybridClock initialized: now 1790570786064380 us; error 32 us; skew 500 ppm
Sep 28 04:46:26 dist-test-slave-j5c3 krb5kdc[17242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570786, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:46:26.067181 18147 init.cc:377] Logged in from keytab as kudu/127.1.226.195@KRBTEST.COM (short username kudu)
I20260928 04:46:26.067636 18147 webserver.cc:460] Webserver started at http://127.1.226.195:35757/ using document root <none> and password file <none>
I20260928 04:46:26.067875 18147 fs_manager.cc:359] Metadata directory not provided
I20260928 04:46:26.067958 18147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:46:26.069468 18147 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:46:26.070188 18162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:46:26.070375 18147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:46:26.070479 18147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "b9b5acc492f8432199d9036830588cec"
format_stamp: "Formatted at 2026-09-28 04:46:05 on dist-test-slave-j5c3"
server_key: "796f9d6088dd4bc5b5eda85f4b6168c4"
server_key_iv: "7950e6942295306e6dc8c2df1a45a47e"
server_key_version: "encryptionkey@0"
I20260928 04:46:26.070832 18147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:46:26.085846 18147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:46:26.088532 18147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:46:26.088701 18147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:46:26.088954 18147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:46:26.089277 18147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:46:26.089318 18147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:46:26.089342 18147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:46:26.089392 18147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:46:26.095259 18147 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:43639
I20260928 04:46:26.095340 18275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:43639 every 8 connection(s)
I20260928 04:46:26.095693 18147 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
Sep 28 04:46:26 dist-test-slave-j5c3 krb5kdc[17242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570786, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:46:26.105163 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 18147
I20260928 04:46:26.105435 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:26.096515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:50897 (local address 127.1.226.254:44913)
0928 04:46:26.096608 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 04:46:26.096610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:46:26.096827 (+ 217us) server_negotiation.cc:407] Connection header received
0928 04:46:26.097345 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:46:26.097347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:46:26.097359 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:46:26.097432 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:46:26.098405 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:26.098898 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:26.099876 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:26.100068 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:26.100127 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:26.102433 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:46:26.102434 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:46:26.102435 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:46:26.102441 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:46:26.103908 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:46:26.104432 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:46:26.104433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:46:26.104434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:46:26.104495 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:46:26.104756 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:46:26.104757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:46:26.104757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:46:26.104874 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:46:26.104949 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 04:46:26.105246 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 04:46:26.105345 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 04:46:26.105664 18278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:26.096497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.195:50897)
0928 04:46:26.096680 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:26.096726 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 04:46:26.097228 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:46:26.097491 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:46:26.097495 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:46:26.098085 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:46:26.098287 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:26.098292 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:26.098982 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:26.098983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:26.099786 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:26.099788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:26.100190 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:26.100191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:26.100318 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:26.100797 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:46:26.100805 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:46:26.102309 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:46:26.104011 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:46:26.104019 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:46:26.104022 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:46:26.104341 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:46:26.104562 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:46:26.104563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:46:26.104563 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:46:26.104686 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:46:26.104941 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:46:26.104948 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:46:26.105099 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 04:46:26.105204 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 04:46:26.105381 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":98,"threads_started":1}
I20260928 04:46:26.106218 18276 heartbeater.cc:347] Connected to a master server at 127.1.226.254:44913
I20260928 04:46:26.106345 18276 heartbeater.cc:464] Registering TS with master...
I20260928 04:46:26.106534 18276 heartbeater.cc:511] Master 127.1.226.254:44913 requested a full tablet report, sending...
I20260928 04:46:26.106989 17818 ts_manager.cc:194] Registered new tserver with Master: b9b5acc492f8432199d9036830588cec (127.1.226.195:43639)
I20260928 04:46:26.107363 17818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.195@KRBTEST.COM'} at 127.1.226.195:50897
I20260928 04:46:26.111948 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:26.105832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53926 (local address 127.1.226.254:44913)
0928 04:46:26.105870 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 04:46:26.105871 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:46:26.105880 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:46:26.105953 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:46:26.105953 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:46:26.105964 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:46:26.106022 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:46:26.106651 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:26.107315 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:26.108138 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:26.108367 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:26.108452 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:26.109602 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:46:26.109603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:46:26.109603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:46:26.109611 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:46:26.110781 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:46:26.111141 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:46:26.111142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:46:26.111143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:46:26.111197 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:46:26.111401 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:46:26.111402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:46:26.111403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:46:26.111513 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:46:26.111708 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 04:46:26.111745 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 04:46:26.111818 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:46:26.112769 1931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:46:26.115399 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:26.113123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53934 (local address 127.1.226.254:44913)
0928 04:46:26.113172 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 04:46:26.113173 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:46:26.113286 (+ 113us) server_negotiation.cc:407] Connection header received
0928 04:46:26.113331 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:46:26.113332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:46:26.113366 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:46:26.113424 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:46:26.113499 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:26.113954 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:26.114774 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:26.114971 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:26.115024 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:26.115173 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:46:26.115218 (+ 45us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 04:46:26.115331 (+ 113us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53934: token expired
Metrics: {"server-negotiator.queue_time_us":28}
W20260928 04:46:26.115455 18009 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53934: token expired
W20260928 04:46:26.115573 17749 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.226.254:44913)
I20260928 04:46:26.120875 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:26.115784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53950 (local address 127.1.226.254:44913)
0928 04:46:26.115817 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 04:46:26.115819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:46:26.115853 (+ 34us) server_negotiation.cc:407] Connection header received
0928 04:46:26.115895 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:46:26.115896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:46:26.115903 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:46:26.115947 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:46:26.116427 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:26.116882 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:26.117521 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:26.117682 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:26.117728 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:26.118858 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:46:26.118859 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:46:26.118859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:46:26.118864 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:46:26.120016 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:46:26.120365 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:46:26.120366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:46:26.120366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:46:26.120413 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:46:26.120535 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:46:26.120536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:46:26.120536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:46:26.120620 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:46:26.120683 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 04:46:26.120744 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 04:46:26.120800 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 04:46:26.121960 17818 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:46:26.123477 17818 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:46:26.126004 17940 tablet_service.cc:1467] Processing CreateTablet for tablet 2d899c84587d4bbaa119a6e2e8568de5 (DEFAULT_TABLE table=test-table [id=26407bcd22ab48df887ff817ac5c450f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:46:26.126204 17940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d899c84587d4bbaa119a6e2e8568de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:46:26.127465 17898 tablet_bootstrap.cc:492] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda: Bootstrap starting.
I20260928 04:46:26.127784 17898 tablet_bootstrap.cc:654] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda: Neither blocks nor log segments found. Creating new log.
I20260928 04:46:26.128623 17898 tablet_bootstrap.cc:492] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda: No bootstrap required, opened a new log
I20260928 04:46:26.128697 17898 ts_tablet_manager.cc:1397] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:46:26.128858 17898 raft_consensus.cc:348] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } }
I20260928 04:46:26.128934 17898 raft_consensus.cc:374] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:46:26.128986 17898 raft_consensus.cc:729] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c8dc7ae435f4dc4a223d2996000aeda, State: Initialized, Role: FOLLOWER
I20260928 04:46:26.129033 17898 consensus_queue.cc:260] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } }
I20260928 04:46:26.129130 17898 ts_tablet_manager.cc:1428] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:46:26.129595 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:26.125591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:43639 (local address 127.0.0.1:38552)
0928 04:46:26.125763 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:26.125773 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:46:26.125815 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:46:26.126133 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:46:26.126134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:46:26.126136 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:46:26.126192 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:26.126193 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:26.127413 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:26.127414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:26.128429 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:26.128432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:26.129394 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:26.129405 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:26.129453 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:26.129469 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 04:46:26.129514 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 04:46:26.129545 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":54,"threads_started":1}
I20260928 04:46:26.129678 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:26.125683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38552 (local address 127.1.226.195:43639)
0928 04:46:26.125934 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 04:46:26.125938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:46:26.125945 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:46:26.125979 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:46:26.126000 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:46:26.126018 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:46:26.126128 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:46:26.126258 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:26.127326 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:26.128536 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:26.129270 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:26.129387 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:26.129467 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 04:46:26.129530 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 04:46:26.129580 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":100,"threads_started":1}
I20260928 04:46:26.130334 18210 tablet_service.cc:1467] Processing CreateTablet for tablet 2d899c84587d4bbaa119a6e2e8568de5 (DEFAULT_TABLE table=test-table [id=26407bcd22ab48df887ff817ac5c450f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:46:26.130684 18210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d899c84587d4bbaa119a6e2e8568de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:46:26.130954 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:26.125495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:35391 (local address 127.0.0.1:48482)
0928 04:46:26.125566 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:26.125580 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:46:26.125628 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:46:26.126031 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:46:26.126032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:46:26.126034 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:46:26.126083 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:26.126084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:26.128399 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:26.128399 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:26.129503 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:26.129506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:26.130689 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:26.130690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:26.130779 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:26.130784 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:46:26.130845 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 04:46:26.130887 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 04:46:26.130980 18282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:26.125628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48482 (local address 127.1.226.194:35391)
0928 04:46:26.125853 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 04:46:26.125858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:46:26.125868 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:46:26.125909 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:46:26.125911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:46:26.125926 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:46:26.126013 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:46:26.126169 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:26.127304 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:26.129902 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:26.130613 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:26.130693 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:26.130747 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 04:46:26.130862 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 04:46:26.130904 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":102,"threads_started":1}
I20260928 04:46:26.131685 18078 tablet_service.cc:1467] Processing CreateTablet for tablet 2d899c84587d4bbaa119a6e2e8568de5 (DEFAULT_TABLE table=test-table [id=26407bcd22ab48df887ff817ac5c450f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:46:26.132004 18078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d899c84587d4bbaa119a6e2e8568de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:46:26.133749 18285 tablet_bootstrap.cc:492] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec: Bootstrap starting.
I20260928 04:46:26.134531 18285 tablet_bootstrap.cc:654] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec: Neither blocks nor log segments found. Creating new log.
I20260928 04:46:26.134719 18286 tablet_bootstrap.cc:492] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5: Bootstrap starting.
I20260928 04:46:26.135138 18285 log.cc:824] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec: Log is configured to *not* fsync() on all Append() calls
I20260928 04:46:26.135251 18286 tablet_bootstrap.cc:654] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5: Neither blocks nor log segments found. Creating new log.
I20260928 04:46:26.135694 18286 log.cc:824] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5: Log is configured to *not* fsync() on all Append() calls
I20260928 04:46:26.135896 18285 tablet_bootstrap.cc:492] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec: No bootstrap required, opened a new log
I20260928 04:46:26.135975 18285 ts_tablet_manager.cc:1397] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:46:26.136401 18286 tablet_bootstrap.cc:492] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5: No bootstrap required, opened a new log
I20260928 04:46:26.136492 18286 ts_tablet_manager.cc:1397] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:46:26.137687 18285 raft_consensus.cc:348] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } }
I20260928 04:46:26.137837 18285 raft_consensus.cc:374] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:46:26.137861 18285 raft_consensus.cc:729] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9b5acc492f8432199d9036830588cec, State: Initialized, Role: FOLLOWER
I20260928 04:46:26.138002 18285 consensus_queue.cc:260] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } }
I20260928 04:46:26.138296 18285 ts_tablet_manager.cc:1428] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 04:46:26.138326 18276 heartbeater.cc:503] Master 127.1.226.254:44913 was elected leader, sending a full tablet report...
I20260928 04:46:26.138245 18286 raft_consensus.cc:348] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } }
I20260928 04:46:26.138438 18286 raft_consensus.cc:374] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:46:26.138481 18286 raft_consensus.cc:729] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70efcba756114c0a9d42decf0be343e5, State: Initialized, Role: FOLLOWER
I20260928 04:46:26.138588 18286 consensus_queue.cc:260] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } }
I20260928 04:46:26.138902 18286 ts_tablet_manager.cc:1428] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 04:46:26.138908 18144 heartbeater.cc:503] Master 127.1.226.254:44913 was elected leader, sending a full tablet report...
I20260928 04:46:26.141945 18290 raft_consensus.cc:482] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:46:26.142040 18290 raft_consensus.cc:504] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } }
I20260928 04:46:26.142369 18290 leader_election.cc:288] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9b5acc492f8432199d9036830588cec (127.1.226.195:43639), 9c8dc7ae435f4dc4a223d2996000aeda (127.1.226.193:36515)
I20260928 04:46:26.145287 18146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:26.142565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:43639 (local address 127.1.226.194:48177)
0928 04:46:26.142621 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:26.142633 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:46:26.142667 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:46:26.142842 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:46:26.142843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:46:26.142845 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:46:26.142917 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:26.142919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:26.143574 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:26.143575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:26.144405 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:26.144407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:26.145091 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:26.145092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:26.145153 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:26.145156 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:46:26.145202 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 04:46:26.145232 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 04:46:26.145320 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:26.142608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:48177 (local address 127.1.226.195:43639)
0928 04:46:26.142646 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 04:46:26.142648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:46:26.142677 (+ 29us) server_negotiation.cc:407] Connection header received
0928 04:46:26.142729 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:46:26.142730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:46:26.142740 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:46:26.142795 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:46:26.142992 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:26.143482 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:26.144491 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:26.145016 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:26.145088 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:26.145097 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 04:46:26.145220 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 04:46:26.145254 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 04:46:26.145701 18230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d899c84587d4bbaa119a6e2e8568de5" candidate_uuid: "70efcba756114c0a9d42decf0be343e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9b5acc492f8432199d9036830588cec" is_pre_election: true
I20260928 04:46:26.145946 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:26.142759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:53931 (local address 127.1.226.193:36515)
0928 04:46:26.142809 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 04:46:26.142811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:46:26.142944 (+ 133us) server_negotiation.cc:407] Connection header received
0928 04:46:26.143009 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:46:26.143010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:46:26.143034 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:46:26.143097 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:46:26.143250 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:26.143759 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:26.144869 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:26.145516 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:26.145599 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:26.145676 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 04:46:26.145845 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 04:46:26.145885 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:46:26.145943 18291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:26.142696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:36515 (local address 127.1.226.194:53931)
0928 04:46:26.142866 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:26.142876 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:46:26.142956 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:46:26.143107 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:46:26.143108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:46:26.143109 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:46:26.143174 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:26.143176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:26.143848 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:26.143849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:26.144787 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:26.144790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:26.145702 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:26.145703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:26.145759 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:26.145762 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:46:26.145828 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 04:46:26.145866 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":3,"thread_start_us":115,"threads_started":1}
I20260928 04:46:26.145967 18230 raft_consensus.cc:2454] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70efcba756114c0a9d42decf0be343e5 in term 0.
I20260928 04:46:26.146238 18033 leader_election.cc:302] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70efcba756114c0a9d42decf0be343e5, b9b5acc492f8432199d9036830588cec; no voters:
I20260928 04:46:26.146377 18290 raft_consensus.cc:2781] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:46:26.146447 18290 raft_consensus.cc:482] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:46:26.146471 18290 raft_consensus.cc:3037] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:46:26.146522 17960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d899c84587d4bbaa119a6e2e8568de5" candidate_uuid: "70efcba756114c0a9d42decf0be343e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" is_pre_election: true
I20260928 04:46:26.146636 17960 raft_consensus.cc:2454] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70efcba756114c0a9d42decf0be343e5 in term 0.
I20260928 04:46:26.147302 18290 raft_consensus.cc:504] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } }
I20260928 04:46:26.147433 18290 leader_election.cc:288] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5 [CANDIDATE]: Term 1 election: Requested vote from peers b9b5acc492f8432199d9036830588cec (127.1.226.195:43639), 9c8dc7ae435f4dc4a223d2996000aeda (127.1.226.193:36515)
I20260928 04:46:26.147601 17960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d899c84587d4bbaa119a6e2e8568de5" candidate_uuid: "70efcba756114c0a9d42decf0be343e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c8dc7ae435f4dc4a223d2996000aeda"
I20260928 04:46:26.147607 18230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d899c84587d4bbaa119a6e2e8568de5" candidate_uuid: "70efcba756114c0a9d42decf0be343e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9b5acc492f8432199d9036830588cec"
I20260928 04:46:26.147687 18230 raft_consensus.cc:3037] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:46:26.147706 17960 raft_consensus.cc:3037] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:46:26.148480 18230 raft_consensus.cc:2454] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70efcba756114c0a9d42decf0be343e5 in term 1.
I20260928 04:46:26.148589 17960 raft_consensus.cc:2454] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70efcba756114c0a9d42decf0be343e5 in term 1.
I20260928 04:46:26.148641 18033 leader_election.cc:302] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70efcba756114c0a9d42decf0be343e5, b9b5acc492f8432199d9036830588cec; no voters:
I20260928 04:46:26.148717 18290 raft_consensus.cc:2781] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:46:26.148891 18290 raft_consensus.cc:686] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5 [term 1 LEADER]: Becoming Leader. State: Replica: 70efcba756114c0a9d42decf0be343e5, State: Running, Role: LEADER
I20260928 04:46:26.149005 18290 consensus_queue.cc:237] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } }
I20260928 04:46:26.149912 17818 catalog_manager.cc:5495] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70efcba756114c0a9d42decf0be343e5 (127.1.226.194). New cstate: current_term: 1 leader_uuid: "70efcba756114c0a9d42decf0be343e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } health_report { overall_health: UNKNOWN } } }
I20260928 04:46:26.160666 18282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:26.158508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48498 (local address 127.1.226.194:35391)
0928 04:46:26.158553 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:46:26.158555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:46:26.158563 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:46:26.158637 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:46:26.158638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:46:26.158653 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:46:26.158718 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:46:26.158797 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:26.159265 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:26.159983 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:26.160217 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:26.160277 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:26.160382 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:46:26.160490 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:46:26.160547 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 04:46:26.160577 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 04:46:26.160594 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:46:26.162436 18230 raft_consensus.cc:1261] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec [term 1 FOLLOWER]: Refusing update from remote peer 70efcba756114c0a9d42decf0be343e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:46:26.162437 17960 raft_consensus.cc:1261] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda [term 1 FOLLOWER]: Refusing update from remote peer 70efcba756114c0a9d42decf0be343e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:46:26.162809 18292 consensus_queue.cc:1035] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:46:26.162941 18290 consensus_queue.cc:1035] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:46:26.164970 18295 mvcc.cc:205] Tried to move back new op lower bound from 7334177940118818816 to 7334177940067184640. Current Snapshot: MvccSnapshot[applied={T|T < 7334177940118818816}]
I20260928 04:46:26.166263 18297 mvcc.cc:205] Tried to move back new op lower bound from 7334177940118818816 to 7334177940067184640. Current Snapshot: MvccSnapshot[applied={T|T < 7334177940118818816}]
I20260928 04:46:26.167148 17818 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:46:26.167233 17818 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:46:26.168272 17818 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504: Sending DeleteTablet for 3 replicas of tablet 2d899c84587d4bbaa119a6e2e8568de5
I20260928 04:46:26.168628 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 17856
I20260928 04:46:26.168684 18078 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d899c84587d4bbaa119a6e2e8568de5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:46:26 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:48482
I20260928 04:46:26.169189 18210 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d899c84587d4bbaa119a6e2e8568de5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:46:26 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:38552
I20260928 04:46:26.169430 18306 tablet_replica.cc:330] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec: stopping tablet replica
I20260928 04:46:26.169523 18306 raft_consensus.cc:2229] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:46:26.169598 18306 raft_consensus.cc:2258] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:46:26.170009 18306 ts_tablet_manager.cc:1905] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:46:26.172122 18306 ts_tablet_manager.cc:1918] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:46:26.172273 18306 log.cc:1196] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/wal/wals/2d899c84587d4bbaa119a6e2e8568de5
I20260928 04:46:26.172583 18306 ts_tablet_manager.cc:1939] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec: Deleting consensus metadata
I20260928 04:46:26.173002 17804 catalog_manager.cc:4841] TS b9b5acc492f8432199d9036830588cec (127.1.226.195:43639): tablet 2d899c84587d4bbaa119a6e2e8568de5 (table test-table [id=26407bcd22ab48df887ff817ac5c450f]) successfully deleted
W20260928 04:46:26.176600 17804 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.226.193:36515 (error 108)
I20260928 04:46:26.177045 18305 tablet_replica.cc:330] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5: stopping tablet replica
I20260928 04:46:26.177068 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 18011
I20260928 04:46:26.177665 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:26.176944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:36515 (local address 127.0.0.1:52008)
0928 04:46:26.177578 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:26.177625 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.193:36515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":610}
W20260928 04:46:26.177820 17804 catalog_manager.cc:4584] TS 9c8dc7ae435f4dc4a223d2996000aeda (127.1.226.193:36515): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d899c84587d4bbaa119a6e2e8568de5: Network error: Client connection negotiation failed: client connection to 127.1.226.193:36515: connect: Connection refused (error 111)
I20260928 04:46:26.182379 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:26.182244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:35391 (local address 127.0.0.1:48504)
0928 04:46:26.182296 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:26.182329 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.194:35391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 04:46:26.182497 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 18147
I20260928 04:46:26.187704 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 17785
I20260928 04:46:46.193302 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:44913
--webserver_interface=127.1.226.254
--webserver_port=45277
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:44913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:46:46.275939 18308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:46:46.276240 18308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:46:46.276291 18308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:46:46.278149 18308 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 04:46:46.278213 18308 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 04:46:46.278236 18308 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 04:46:46.278264 18308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:46:46.278299 18308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:46:46.278331 18308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:46:46.278365 18308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:46:46.278399 18308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:46:46.280025 18308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:44913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:44913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=45277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:46:46.280337 18308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:46:46.280598 18308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:46:46.285983 18316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:46:46.285983 18314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:46:46.286051 18308 server_base.cc:1034] running on GCE node
W20260928 04:46:46.285989 18313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:46:46.286504 18308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:46:46.286859 18308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:46:46.288043 18308 hybrid_clock.cc:648] HybridClock initialized: now 1790570806288033 us; error 34 us; skew 500 ppm
Sep 28 04:46:46 dist-test-slave-j5c3 krb5kdc[17242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570806, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:46:46.290917 18308 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:46:46.291419 18308 webserver.cc:460] Webserver started at http://127.1.226.254:45277/ using document root <none> and password file <none>
I20260928 04:46:46.291662 18308 fs_manager.cc:359] Metadata directory not provided
I20260928 04:46:46.291734 18308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:46:46.293327 18308 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:46:46.294155 18323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:46:46.294396 18308 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:46:46.294515 18308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "7de1202fbe4b4f80bdb46a2663c75504"
format_stamp: "Formatted at 2026-09-28 04:46:04 on dist-test-slave-j5c3"
server_key: "cb44743a5d0eed75b4d08779cfd9454a"
server_key_iv: "b4cd6b9ca4fceb692159fcaac1aa626f"
server_key_version: "encryptionkey@0"
I20260928 04:46:46.294900 18308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:46:46.308379 18308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:46:46.310379 18308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:46:46.310544 18308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:46:46.314688 18308 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:44913
I20260928 04:46:46.314729 18375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:44913 every 8 connection(s)
I20260928 04:46:46.315143 18308 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:46:46.316675 18376 sys_catalog.cc:263] Verifying existing consensus state
I20260928 04:46:46.317265 18376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504: Bootstrap starting.
I20260928 04:46:46.318490 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 18308
I20260928 04:46:46.318768 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:36515
--local_ip_for_outbound_sockets=127.1.226.193
--tserver_master_addrs=127.1.226.254:44913
--webserver_port=32933
--webserver_interface=127.1.226.193
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:46:46.326164 18376 log.cc:824] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504: Log is configured to *not* fsync() on all Append() calls
I20260928 04:46:46.328608 18376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:46:46.328938 18376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504: Bootstrap complete.
I20260928 04:46:46.331228 18376 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de1202fbe4b4f80bdb46a2663c75504" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44913 } }
I20260928 04:46:46.331504 18376 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7de1202fbe4b4f80bdb46a2663c75504, State: Initialized, Role: FOLLOWER
I20260928 04:46:46.331719 18376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de1202fbe4b4f80bdb46a2663c75504" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44913 } }
I20260928 04:46:46.331820 18376 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:46:46.331874 18376 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:46:46.331923 18376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [term 2 FOLLOWER]: Advancing to term 3
I20260928 04:46:46.333041 18376 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de1202fbe4b4f80bdb46a2663c75504" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44913 } }
I20260928 04:46:46.333164 18376 leader_election.cc:302] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7de1202fbe4b4f80bdb46a2663c75504; no voters:
I20260928 04:46:46.333355 18376 leader_election.cc:288] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 04:46:46.333432 18380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 04:46:46.333636 18376 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:46:46.333699 18380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [term 3 LEADER]: Becoming Leader. State: Replica: 7de1202fbe4b4f80bdb46a2663c75504, State: Running, Role: LEADER
I20260928 04:46:46.333971 18380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de1202fbe4b4f80bdb46a2663c75504" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44913 } }
I20260928 04:46:46.334312 18381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7de1202fbe4b4f80bdb46a2663c75504" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de1202fbe4b4f80bdb46a2663c75504" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44913 } } }
I20260928 04:46:46.334406 18381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [sys.catalog]: This master's current role is: LEADER
I20260928 04:46:46.334622 18382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7de1202fbe4b4f80bdb46a2663c75504. Latest consensus state: current_term: 3 leader_uuid: "7de1202fbe4b4f80bdb46a2663c75504" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7de1202fbe4b4f80bdb46a2663c75504" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44913 } } }
I20260928 04:46:46.334769 18382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504 [sys.catalog]: This master's current role is: LEADER
I20260928 04:46:46.334949 18389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:46:46.335999 18389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:46:46.336103 18389 catalog_manager.cc:1259] Loaded cluster ID: fa05fa53151e461ca22e53a29d32bbe1
I20260928 04:46:46.336136 18389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:46:46.337280 18389 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:46:46.337471 18389 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504: Loaded TSK: 0
I20260928 04:46:46.337520 18389 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7de1202fbe4b4f80bdb46a2663c75504: Loaded TSK: 1
F20260928 04:46:46.337760 18389 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fece1efa9ad google::LogMessage::Fail() at ??:0
@ 0x7fece1efd88a google::LogMessage::SendToLog() at ??:0
@ 0x7fece1efa4de google::LogMessage::Flush() at ??:0
@ 0x7fece1efba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fece4667104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fece4664d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fece9e541cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fece9e5448e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fece9e54594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fece9e1ac48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fece9e24ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fece3dc80d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fece3dbd954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fece4c216db start_thread at ??:0
@ 0x7fece09f861f clone at ??:0
W20260928 04:46:46.426559 18378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:46:46.426767 18378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:46:46.426823 18378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:46:46.428020 18378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:46:46.428064 18378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:46:46.428092 18378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:46:46.429625 18378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:36515
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=32933
--tserver_master_addrs=127.1.226.254:44913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:46:46.429883 18378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:46:46.430137 18378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:46:46.435405 18418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:46:46.435499 18417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:46:46.435405 18420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:46:46.435868 18378 server_base.cc:1034] running on GCE node
I20260928 04:46:46.436136 18378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:46:46.436427 18378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:46:46.437597 18378 hybrid_clock.cc:648] HybridClock initialized: now 1790570806437574 us; error 41 us; skew 500 ppm
Sep 28 04:46:46 dist-test-slave-j5c3 krb5kdc[17242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570806, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:46:46.440442 18378 init.cc:377] Logged in from keytab as kudu/127.1.226.193@KRBTEST.COM (short username kudu)
I20260928 04:46:46.440938 18378 webserver.cc:460] Webserver started at http://127.1.226.193:32933/ using document root <none> and password file <none>
I20260928 04:46:46.441176 18378 fs_manager.cc:359] Metadata directory not provided
I20260928 04:46:46.441246 18378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:46:46.442659 18378 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:46:46.443498 18427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:46:46.443673 18378 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:46:46.443774 18378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "9c8dc7ae435f4dc4a223d2996000aeda"
format_stamp: "Formatted at 2026-09-28 04:46:05 on dist-test-slave-j5c3"
server_key: "6555c58716ad430b02df54ea4ec3b1ff"
server_key_iv: "c1e73899d9b3de377b1a75a6f446a72b"
server_key_version: "encryptionkey@0"
I20260928 04:46:46.444197 18378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:46:46.453547 18378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:46:46.455966 18378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:46:46.456140 18378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:46:46.456434 18378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:46:46.456889 18434 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 04:46:46.458915 18378 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 04:46:46.458987 18378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 04:46:46.459033 18378 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 04:46:46.459690 18378 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 04:46:46.459749 18378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 04:46:46.459829 18434 tablet_bootstrap.cc:492] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda: Bootstrap starting.
I20260928 04:46:46.467238 18378 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:36515
I20260928 04:46:46.467682 18378 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
I20260928 04:46:46.468351 18541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:36515 every 8 connection(s)
I20260928 04:46:46.471356 18434 log.cc:824] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda: Log is configured to *not* fsync() on all Append() calls
I20260928 04:46:46.474339 18434 tablet_bootstrap.cc:492] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:46:46.474704 18434 tablet_bootstrap.cc:492] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda: Bootstrap complete.
I20260928 04:46:46.475934 18434 ts_tablet_manager.cc:1397] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.000s
I20260928 04:46:46.477653 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 18378
I20260928 04:46:46.477516 18434 raft_consensus.cc:348] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } }
I20260928 04:46:46.477885 18434 raft_consensus.cc:729] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c8dc7ae435f4dc4a223d2996000aeda, State: Initialized, Role: FOLLOWER
I20260928 04:46:46.477967 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:35391
--local_ip_for_outbound_sockets=127.1.226.194
--tserver_master_addrs=127.1.226.254:44913
--webserver_port=45573
--webserver_interface=127.1.226.194
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:46:46.478034 18434 consensus_queue.cc:260] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } }
I20260928 04:46:46.478318 18434 ts_tablet_manager.cc:1428] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda: Time spent starting tablet: real 0.002s user 0.006s sys 0.000s
W20260928 04:46:46.540594 18544 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:46:46.468482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.193:41423)
0928 04:46:46.468677 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:46.468723 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 04:46:46.469237 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:46:46.540413 (+ 71176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":118,"threads_started":1}
W20260928 04:46:46.540865 18542 heartbeater.cc:644] Failed to heartbeat to 127.1.226.254:44913 (0 consecutive failures): Network error: Failed to ping master at 127.1.226.254:44913: Client connection negotiation failed: client connection to 127.1.226.254:44913: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 04:46:46.541244 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:46.541128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.193:55883)
0928 04:46:46.541183 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:46.541224 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 04:46:46.541623 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:46.541528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.193:38251)
0928 04:46:46.541577 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:46.541607 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 04:46:46.541740 18542 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 04:46:46.563565 18547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:46:46.563809 18547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:46:46.563851 18547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:46:46.565084 18547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:46:46.565150 18547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:46:46.565196 18547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:46:46.566677 18547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:35391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=45573
--tserver_master_addrs=127.1.226.254:44913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:46:46.566944 18547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:46:46.567181 18547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:46:46.572480 18555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:46:46.572465 18553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:46:46.572458 18552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:46:46.572499 18547 server_base.cc:1034] running on GCE node
I20260928 04:46:46.572873 18547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:46:46.573136 18547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:46:46.574327 18547 hybrid_clock.cc:648] HybridClock initialized: now 1790570806574318 us; error 40 us; skew 500 ppm
Sep 28 04:46:46 dist-test-slave-j5c3 krb5kdc[17242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570806, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:46:46.577085 18547 init.cc:377] Logged in from keytab as kudu/127.1.226.194@KRBTEST.COM (short username kudu)
I20260928 04:46:46.577510 18547 webserver.cc:460] Webserver started at http://127.1.226.194:45573/ using document root <none> and password file <none>
I20260928 04:46:46.577702 18547 fs_manager.cc:359] Metadata directory not provided
I20260928 04:46:46.577744 18547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:46:46.579056 18547 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:46:46.579717 18562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:46:46.579912 18547 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:46:46.580013 18547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "70efcba756114c0a9d42decf0be343e5"
format_stamp: "Formatted at 2026-09-28 04:46:05 on dist-test-slave-j5c3"
server_key: "20ef71b86b9269b3908bb4e546a98aea"
server_key_iv: "edc59b9ba38ed4e4fdc2a4f9a604cc57"
server_key_version: "encryptionkey@0"
I20260928 04:46:46.580456 18547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:46:46.593667 18547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:46:46.595621 18547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:46:46.595789 18547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:46:46.596041 18547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:46:46.596525 18569 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 04:46:46.597990 18547 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 04:46:46.598048 18547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 04:46:46.598111 18547 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 04:46:46.598758 18547 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 04:46:46.598816 18547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 04:46:46.598874 18569 tablet_bootstrap.cc:492] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5: Bootstrap starting.
I20260928 04:46:46.605921 18547 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:35391
I20260928 04:46:46.606091 18676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:35391 every 8 connection(s)
I20260928 04:46:46.606388 18547 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
I20260928 04:46:46.607635 18679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:46.607096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.194:43197)
0928 04:46:46.607289 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:46.607416 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":81,"threads_started":1}
W20260928 04:46:46.607935 18677 heartbeater.cc:644] Failed to heartbeat to 127.1.226.254:44913 (0 consecutive failures): Network error: Failed to ping master at 127.1.226.254:44913: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
I20260928 04:46:46.608279 18679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:46.608138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.194:43809)
0928 04:46:46.608216 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:46.608247 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 04:46:46.608644 18679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:46.608545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.194:42465)
0928 04:46:46.608598 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:46.608622 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 04:46:46.608817 18677 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 04:46:46.610018 18569 log.cc:824] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5: Log is configured to *not* fsync() on all Append() calls
I20260928 04:46:46.612232 18569 tablet_bootstrap.cc:492] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:46:46.612527 18569 tablet_bootstrap.cc:492] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5: Bootstrap complete.
I20260928 04:46:46.613462 18569 ts_tablet_manager.cc:1397] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20260928 04:46:46.613539 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 18547
I20260928 04:46:46.613821 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:43639
--local_ip_for_outbound_sockets=127.1.226.195
--tserver_master_addrs=127.1.226.254:44913
--webserver_port=35757
--webserver_interface=127.1.226.195
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:46:46.614820 18569 raft_consensus.cc:348] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } }
I20260928 04:46:46.615242 18569 raft_consensus.cc:729] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70efcba756114c0a9d42decf0be343e5, State: Initialized, Role: FOLLOWER
I20260928 04:46:46.615386 18569 consensus_queue.cc:260] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } }
I20260928 04:46:46.615664 18569 ts_tablet_manager.cc:1428] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 04:46:46.615963 18679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:46.615824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.194:55215)
0928 04:46:46.615898 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:46.615929 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 04:46:46.616276 18679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:46.616210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.194:42645)
0928 04:46:46.616236 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:46.616257 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260928 04:46:46.699697 18681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:46:46.699941 18681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:46:46.699968 18681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:46:46.701260 18681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:46:46.701377 18681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:46:46.701450 18681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:46:46.703032 18681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:43639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=35757
--tserver_master_addrs=127.1.226.254:44913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:46:46.703306 18681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:46:46.703577 18681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:46:46.708886 18688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:46:46.708915 18687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:46:46.708997 18690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:46:46.708998 18681 server_base.cc:1034] running on GCE node
I20260928 04:46:46.709400 18681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:46:46.709676 18681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:46:46.710881 18681 hybrid_clock.cc:648] HybridClock initialized: now 1790570806710868 us; error 31 us; skew 500 ppm
Sep 28 04:46:46 dist-test-slave-j5c3 krb5kdc[17242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570806, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:46:46.713632 18681 init.cc:377] Logged in from keytab as kudu/127.1.226.195@KRBTEST.COM (short username kudu)
I20260928 04:46:46.714121 18681 webserver.cc:460] Webserver started at http://127.1.226.195:35757/ using document root <none> and password file <none>
I20260928 04:46:46.714316 18681 fs_manager.cc:359] Metadata directory not provided
I20260928 04:46:46.714358 18681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:46:46.715698 18681 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:46:46.716439 18697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:46:46.716642 18681 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:46:46.716734 18681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "b9b5acc492f8432199d9036830588cec"
format_stamp: "Formatted at 2026-09-28 04:46:05 on dist-test-slave-j5c3"
server_key: "796f9d6088dd4bc5b5eda85f4b6168c4"
server_key_iv: "7950e6942295306e6dc8c2df1a45a47e"
server_key_version: "encryptionkey@0"
I20260928 04:46:46.717096 18681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:46:46.734022 18681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:46:46.736604 18681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:46:46.736729 18681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:46:46.736943 18681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:46:46.737293 18681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:46:46.737329 18681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:46:46.737355 18681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:46:46.737403 18681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:46:46.743985 18681 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:43639
I20260928 04:46:46.744072 18810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:43639 every 8 connection(s)
I20260928 04:46:46.744423 18681 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
I20260928 04:46:46.745615 18813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:46.745210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.195:34221)
0928 04:46:46.745382 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:46.745461 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
W20260928 04:46:46.746062 18811 heartbeater.cc:644] Failed to heartbeat to 127.1.226.254:44913 (0 consecutive failures): Network error: Failed to ping master at 127.1.226.254:44913: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
I20260928 04:46:46.746397 18813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:46.746299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.195:52933)
0928 04:46:46.746348 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:46.746375 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 04:46:46.746667 18813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:46.746618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.195:38119)
0928 04:46:46.746639 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:46.746656 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260928 04:46:46.746778 18811 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 04:46:46.749527 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 18681
I20260928 04:46:47.542405 18815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:47.542170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.193:50415)
0928 04:46:47.542297 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:47.542337 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260928 04:46:47.616955 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:47.616710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.194:49795)
0928 04:46:47.616848 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:47.616892 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260928 04:46:47.747356 18817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:47.747093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.195:60811)
0928 04:46:47.747233 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:47.747282 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":56,"threads_started":1}
I20260928 04:46:47.987498 18818 raft_consensus.cc:482] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:46:47.987628 18818 raft_consensus.cc:504] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } }
I20260928 04:46:47.988058 18818 leader_election.cc:288] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 70efcba756114c0a9d42decf0be343e5 (127.1.226.194:35391), b9b5acc492f8432199d9036830588cec (127.1.226.195:43639)
Sep 28 04:46:47 dist-test-slave-j5c3 krb5kdc[17242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570806, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for kudu/127.1.226.195@KRBTEST.COM
Sep 28 04:46:47 dist-test-slave-j5c3 krb5kdc[17242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570806, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for kudu/127.1.226.194@KRBTEST.COM
I20260928 04:46:48.002607 18820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:47.988470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:43639 (local address 127.1.226.193:46783)
0928 04:46:47.988602 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:47.988611 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:46:47.988650 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:46:47.989600 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:46:47.989601 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:46:47.989749 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:46:47.989831 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:47.989833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:47.990966 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:47.990967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:47.992305 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:47.992309 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:47.992936 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:47.992937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:47.993038 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:47.996859 (+ 3821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:46:47.996868 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:46:47.998517 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:46:48.000588 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:46:48.000596 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:46:48.000599 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:46:48.000826 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:46:48.001258 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:46:48.001259 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:46:48.001259 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:46:48.001418 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:46:48.001970 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:46:48.001979 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:46:48.002156 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 04:46:48.002285 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 04:46:48.002482 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":104,"thread_start_us":46,"threads_started":1}
I20260928 04:46:48.002638 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:47.988439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:53191 (local address 127.1.226.194:35391)
0928 04:46:47.988624 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 04:46:47.988635 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 04:46:47.988657 (+ 22us) server_negotiation.cc:407] Connection header received
0928 04:46:47.988761 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:46:47.988763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:46:47.988805 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:46:47.988921 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:46:47.989857 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:47.990876 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:47.992396 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:47.992802 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:47.992925 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:47.998911 (+ 5986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:46:47.998915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:46:47.998915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:46:47.998927 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:46:48.000760 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:46:48.001147 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:46:48.001150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:46:48.001151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:46:48.001264 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:46:48.001537 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:46:48.001538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:46:48.001538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:46:48.001868 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:46:48.001963 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 04:46:48.002379 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 04:46:48.002516 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":66,"threads_started":1}
I20260928 04:46:48.002631 18821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:47.988570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:46783 (local address 127.1.226.195:43639)
0928 04:46:47.988740 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 04:46:47.988749 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:46:47.988779 (+ 30us) server_negotiation.cc:407] Connection header received
0928 04:46:47.989385 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:46:47.989389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:46:47.989457 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:46:47.989605 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:46:47.989939 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:47.990838 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:47.992441 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:47.992822 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:47.992951 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:47.998689 (+ 5738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:46:47.998693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:46:47.998693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:46:47.998703 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:46:48.000487 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:46:48.000947 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:46:48.000951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:46:48.000951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:46:48.001050 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:46:48.001515 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:46:48.001516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:46:48.001517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:46:48.001873 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:46:48.001963 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 04:46:48.002367 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 04:46:48.002515 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":68,"threads_started":1}
I20260928 04:46:48.002666 18815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:47.988294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:35391 (local address 127.1.226.193:53191)
0928 04:46:47.988364 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:47.988380 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 04:46:47.988499 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:46:47.988983 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:46:47.988986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:46:47.989425 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:46:47.989727 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:47.989734 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:47.990979 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:47.990980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:47.992306 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:47.992309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:47.992908 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:47.992909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:47.993039 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:47.996876 (+ 3837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:46:47.996881 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:46:47.998780 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:46:48.000881 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:46:48.000882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:46:48.000882 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:46:48.001059 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:46:48.001337 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:46:48.001338 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:46:48.001338 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:46:48.001443 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:46:48.001970 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:46:48.001979 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:46:48.002156 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 04:46:48.002285 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 04:46:48.002554 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":130}
I20260928 04:46:48.003557 18631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d899c84587d4bbaa119a6e2e8568de5" candidate_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "70efcba756114c0a9d42decf0be343e5" is_pre_election: true
I20260928 04:46:48.003728 18631 raft_consensus.cc:2454] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c8dc7ae435f4dc4a223d2996000aeda in term 1.
W20260928 04:46:48.003938 18430 leader_election.cc:341] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9b5acc492f8432199d9036830588cec (127.1.226.195:43639): Not found: Tablet not found: 2d899c84587d4bbaa119a6e2e8568de5
I20260928 04:46:48.004071 18430 leader_election.cc:302] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 70efcba756114c0a9d42decf0be343e5, 9c8dc7ae435f4dc4a223d2996000aeda; no voters: b9b5acc492f8432199d9036830588cec
I20260928 04:46:48.004240 18818 raft_consensus.cc:2781] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 04:46:48.004323 18818 raft_consensus.cc:482] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:46:48.004384 18818 raft_consensus.cc:3037] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda [term 1 FOLLOWER]: Advancing to term 2
I20260928 04:46:48.005637 18818 raft_consensus.cc:504] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } }
I20260928 04:46:48.005771 18818 leader_election.cc:288] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda [CANDIDATE]: Term 2 election: Requested vote from peers 70efcba756114c0a9d42decf0be343e5 (127.1.226.194:35391), b9b5acc492f8432199d9036830588cec (127.1.226.195:43639)
I20260928 04:46:48.006070 18631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d899c84587d4bbaa119a6e2e8568de5" candidate_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "70efcba756114c0a9d42decf0be343e5"
I20260928 04:46:48.006186 18631 raft_consensus.cc:3037] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5 [term 1 FOLLOWER]: Advancing to term 2
W20260928 04:46:48.006214 18430 leader_election.cc:341] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b9b5acc492f8432199d9036830588cec (127.1.226.195:43639): Not found: Tablet not found: 2d899c84587d4bbaa119a6e2e8568de5
I20260928 04:46:48.007390 18631 raft_consensus.cc:2454] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c8dc7ae435f4dc4a223d2996000aeda in term 2.
I20260928 04:46:48.007668 18430 leader_election.cc:302] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 70efcba756114c0a9d42decf0be343e5, 9c8dc7ae435f4dc4a223d2996000aeda; no voters: b9b5acc492f8432199d9036830588cec
I20260928 04:46:48.007771 18818 raft_consensus.cc:2781] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda [term 2 FOLLOWER]: Leader election won for term 2
I20260928 04:46:48.007911 18818 raft_consensus.cc:686] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda [term 2 LEADER]: Becoming Leader. State: Replica: 9c8dc7ae435f4dc4a223d2996000aeda, State: Running, Role: LEADER
I20260928 04:46:48.008014 18818 consensus_queue.cc:237] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } }
I20260928 04:46:48.008364 18815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:48.008231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.193:36259)
0928 04:46:48.008281 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:48.008311 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 04:46:48.596940 18631 raft_consensus.cc:1261] T 2d899c84587d4bbaa119a6e2e8568de5 P 70efcba756114c0a9d42decf0be343e5 [term 2 FOLLOWER]: Refusing update from remote peer 9c8dc7ae435f4dc4a223d2996000aeda: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260928 04:46:48.597383 18824 consensus_queue.cc:1035] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda [LEADER]: Connected to new peer: Peer: permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260928 04:46:48.598556 18827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:48.598274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.194:60155)
0928 04:46:48.598399 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:48.598513 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260928 04:46:48.599794 18831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:48.599515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.193:45691)
0928 04:46:48.599669 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:48.599724 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"spinlock_wait_cycles":14592,"thread_start_us":134,"threads_started":1}
W20260928 04:46:48.599948 18430 consensus_peers.cc:487] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda -> Peer b9b5acc492f8432199d9036830588cec (127.1.226.195:43639): Couldn't send request to peer b9b5acc492f8432199d9036830588cec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d899c84587d4bbaa119a6e2e8568de5. This is attempt 1: this message will repeat every 5th retry.
I20260928 04:46:48.600267 18827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:48.600141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.194:43901)
0928 04:46:48.600198 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:48.600242 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 04:46:48.748024 18832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:48.747809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.195:57813)
0928 04:46:48.747939 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:48.747982 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260928 04:46:49.131374 18834 ts_tablet_manager.cc:927] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec: Initiating tablet copy from peer 9c8dc7ae435f4dc4a223d2996000aeda (127.1.226.193:36515)
I20260928 04:46:49.131595 18834 tablet_copy_client.cc:316] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec: tablet copy: Beginning tablet copy session from remote peer at address 127.1.226.193:36515
Sep 28 04:46:49 dist-test-slave-j5c3 krb5kdc[17242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570806, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for kudu/127.1.226.193@KRBTEST.COM
I20260928 04:46:49.139434 18836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:49.132047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:38447 (local address 127.1.226.193:36515)
0928 04:46:49.132194 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 04:46:49.132199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:46:49.132212 (+ 13us) server_negotiation.cc:407] Connection header received
0928 04:46:49.132242 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:46:49.132244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:46:49.132267 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:46:49.132340 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:46:49.132900 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:49.133582 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:49.134357 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:46:49.134668 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:46:49.134724 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:49.136908 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:46:49.136919 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:46:49.136919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:46:49.136926 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:46:49.138303 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:46:49.138671 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:46:49.138674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:46:49.138674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:46:49.138738 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:46:49.138902 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:46:49.138903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:46:49.138903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:46:49.139066 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:46:49.139129 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 04:46:49.139280 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 04:46:49.139368 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":46,"threads_started":1}
I20260928 04:46:49.139436 18832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:49.131912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:36515 (local address 127.1.226.195:38447)
0928 04:46:49.131974 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:49.131996 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 04:46:49.132078 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:46:49.132356 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:46:49.132363 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:46:49.132671 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:46:49.132760 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:49.132763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:49.133669 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:49.133670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:49.134281 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:46:49.134283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:46:49.134735 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:46:49.134736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:46:49.134793 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:46:49.135339 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:46:49.135345 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:46:49.136818 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:46:49.138398 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:46:49.138404 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:46:49.138406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:46:49.138598 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:46:49.138795 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:46:49.138796 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:46:49.138796 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:46:49.138848 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:46:49.139122 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:46:49.139127 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:46:49.139174 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 04:46:49.139239 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 04:46:49.139325 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 04:46:49.139922 18516 tablet_copy_service.cc:140] P 9c8dc7ae435f4dc4a223d2996000aeda: Received BeginTabletCopySession request for tablet 2d899c84587d4bbaa119a6e2e8568de5 from peer b9b5acc492f8432199d9036830588cec ({username='kudu', principal='kudu/127.1.226.195@KRBTEST.COM'} at 127.1.226.195:38447)
I20260928 04:46:49.140003 18516 tablet_copy_service.cc:161] P 9c8dc7ae435f4dc4a223d2996000aeda: Beginning new tablet copy session on tablet 2d899c84587d4bbaa119a6e2e8568de5 from peer b9b5acc492f8432199d9036830588cec at {username='kudu', principal='kudu/127.1.226.195@KRBTEST.COM'} at 127.1.226.195:38447: session id = b9b5acc492f8432199d9036830588cec-2d899c84587d4bbaa119a6e2e8568de5
I20260928 04:46:49.140558 18516 tablet_copy_source_session.cc:205] T 2d899c84587d4bbaa119a6e2e8568de5 P 9c8dc7ae435f4dc4a223d2996000aeda: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 04:46:49.141569 18834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d899c84587d4bbaa119a6e2e8568de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:46:49.144501 18834 tablet_copy_client.cc:791] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec: tablet copy: Starting download of 0 data blocks...
I20260928 04:46:49.144673 18834 tablet_copy_client.cc:655] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec: tablet copy: Starting download of 1 WAL segments...
I20260928 04:46:49.145694 18834 tablet_copy_client.cc:531] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 04:46:49.146556 18834 tablet_bootstrap.cc:492] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec: Bootstrap starting.
I20260928 04:46:49.148067 18834 log.cc:824] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec: Log is configured to *not* fsync() on all Append() calls
I20260928 04:46:49.150432 18834 tablet_bootstrap.cc:492] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:46:49.150656 18834 tablet_bootstrap.cc:492] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec: Bootstrap complete.
I20260928 04:46:49.150803 18834 ts_tablet_manager.cc:1397] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 04:46:49.151755 18834 raft_consensus.cc:348] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } }
I20260928 04:46:49.151938 18834 raft_consensus.cc:729] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9b5acc492f8432199d9036830588cec, State: Initialized, Role: FOLLOWER
I20260928 04:46:49.152086 18834 consensus_queue.cc:260] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70efcba756114c0a9d42decf0be343e5" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35391 } } peers { permanent_uuid: "b9b5acc492f8432199d9036830588cec" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 43639 } } peers { permanent_uuid: "9c8dc7ae435f4dc4a223d2996000aeda" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 36515 } }
I20260928 04:46:49.152441 18834 ts_tablet_manager.cc:1428] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260928 04:46:49.152716 18832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:49.152614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.195:49701)
0928 04:46:49.152653 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:49.152683 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 04:46:49.152721 18516 tablet_copy_service.cc:342] P 9c8dc7ae435f4dc4a223d2996000aeda: Request end of tablet copy session b9b5acc492f8432199d9036830588cec-2d899c84587d4bbaa119a6e2e8568de5 received from {username='kudu', principal='kudu/127.1.226.195@KRBTEST.COM'} at 127.1.226.195:38447
I20260928 04:46:49.152776 18516 tablet_copy_service.cc:433] P 9c8dc7ae435f4dc4a223d2996000aeda: ending tablet copy session b9b5acc492f8432199d9036830588cec-2d899c84587d4bbaa119a6e2e8568de5 on tablet 2d899c84587d4bbaa119a6e2e8568de5 with peer b9b5acc492f8432199d9036830588cec
I20260928 04:46:49.153198 18832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:49.153086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.195:38599)
0928 04:46:49.153142 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:49.153173 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 04:46:49.522543 18765 raft_consensus.cc:1203] T 2d899c84587d4bbaa119a6e2e8568de5 P b9b5acc492f8432199d9036830588cec [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260928 04:46:49.522599 18841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:49.522366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.193:55529)
0928 04:46:49.522519 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:49.522558 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20260928 04:46:49.523041 18832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:49.522926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.195:37361)
0928 04:46:49.522981 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:49.523014 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 04:46:49.600919 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:49.600695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.194:44141)
0928 04:46:49.600824 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:49.600869 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":56,"threads_started":1}
I20260928 04:46:50.523285 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:50.523063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.193:41263)
0928 04:46:50.523195 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:50.523236 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260928 04:46:50.523749 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:50.523543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.195:37569)
0928 04:46:50.523662 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:50.523700 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260928 04:46:50.601712 18845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:50.601443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.194:40913)
0928 04:46:50.601612 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:50.601663 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":85,"threads_started":1}
I20260928 04:46:51.524482 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:51.524235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.195:53599)
0928 04:46:51.524393 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:51.524428 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260928 04:46:51.524483 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:51.524205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.193:56259)
0928 04:46:51.524339 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:51.524425 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":59,"threads_started":1}
I20260928 04:46:51.524891 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:51.524752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.193:42231)
0928 04:46:51.524802 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:51.524834 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 04:46:51.602607 18848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:51.602352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.194:53007)
0928 04:46:51.602521 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:51.602560 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":93,"threads_started":1}
I20260928 04:46:52.525336 18849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:52.525105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.195:54981)
0928 04:46:52.525254 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:52.525290 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
I20260928 04:46:52.525552 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:52.525306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.193:47023)
0928 04:46:52.525462 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:52.525500 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":112,"threads_started":1}
I20260928 04:46:52.603320 18851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:52.603087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.194:51099)
0928 04:46:52.603237 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:52.603274 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":63,"threads_started":1}
I20260928 04:46:53.526125 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:53.525893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.195:54095)
0928 04:46:53.526040 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:53.526077 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260928 04:46:53.526225 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:53.525984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.193:49045)
0928 04:46:53.526125 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:53.526157 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
I20260928 04:46:53.604055 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:53.603773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.194:58651)
0928 04:46:53.603968 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:53.604000 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":111,"threads_started":1}
I20260928 04:46:54.526889 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:54.526659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.195:47583)
0928 04:46:54.526795 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:54.526835 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260928 04:46:54.526983 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:54.526716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.193:38371)
0928 04:46:54.526871 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:54.526914 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":101,"threads_started":1}
I20260928 04:46:54.604806 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:54.604531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.194:34539)
0928 04:46:54.604698 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:54.604749 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":67,"threads_started":1}
I20260928 04:46:55.527679 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:55.527440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.195:32999)
0928 04:46:55.527598 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:55.527635 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
I20260928 04:46:55.527762 18859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:55.527484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.193:60099)
0928 04:46:55.527629 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:55.527667 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":89,"threads_started":1}
I20260928 04:46:55.605548 18860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:55.605339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.194:36209)
0928 04:46:55.605471 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:55.605501 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":96,"threads_started":1}
I20260928 04:46:56.528576 18862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:56.528301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.195:52723)
0928 04:46:56.528459 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:56.528509 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":68,"threads_started":1}
I20260928 04:46:56.528568 18861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:56.528305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.193:51655)
0928 04:46:56.528459 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:56.528492 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":65,"threads_started":1}
I20260928 04:46:56.606287 18863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:56.606020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.194:37113)
0928 04:46:56.606196 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:56.606238 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":63,"threads_started":1}
I20260928 04:46:57.529567 18864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:57.529204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.193:56951)
0928 04:46:57.529435 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:57.529487 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":106,"threads_started":1}
I20260928 04:46:57.529603 18865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:57.529307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.195:34525)
0928 04:46:57.529477 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:57.529528 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":65,"threads_started":1}
I20260928 04:46:57.607120 18866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:57.606800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.194:47949)
0928 04:46:57.607035 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:57.607081 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":82,"threads_started":1}
I20260928 04:46:58.530411 18867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:58.530152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.193:37615)
0928 04:46:58.530290 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:58.530338 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20260928 04:46:58.530530 18868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:58.530252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.195:52709)
0928 04:46:58.530412 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:58.530454 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":96,"threads_started":1}
I20260928 04:46:58.607936 18869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:58.607638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.194:53169)
0928 04:46:58.607850 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:58.607897 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":76,"threads_started":1}
I20260928 04:46:59.531237 18871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:59.530986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.193:47539)
0928 04:46:59.531143 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:59.531186 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":87,"threads_started":1}
I20260928 04:46:59.531276 18870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:59.530986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.195:34773)
0928 04:46:59.531192 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:59.531231 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":91,"threads_started":1}
I20260928 04:46:59.608716 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:46:59.608486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.194:47301)
0928 04:46:59.608629 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 04:46:59.608672 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
I20260928 04:47:00.532277 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:00.531945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.195:55375)
0928 04:47:00.532132 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:00.532225 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":86,"threads_started":1}
I20260928 04:47:00.532276 18874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:00.531945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.193:47859)
0928 04:47:00.532118 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:00.532227 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":83,"threads_started":1}
I20260928 04:47:00.609505 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:00.609247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.194:44433)
0928 04:47:00.609390 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:00.609450 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
I20260928 04:47:01.533108 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:01.532858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.193:38187)
0928 04:47:01.533028 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:01.533063 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":74,"threads_started":1}
I20260928 04:47:01.533128 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:01.532858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.195:46609)
0928 04:47:01.533043 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:01.533086 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":76,"threads_started":1}
I20260928 04:47:01.610397 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:01.610121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44913 (local address 127.1.226.194:59395)
0928 04:47:01.610264 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:01.610302 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.254:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260928 04:47:01.758534 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 18378
I20260928 04:47:01.763384 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 18547
I20260928 04:47:01.768033 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 18681
2026-09-28T04:47:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58219 ms)
[----------] 30 tests from SecurityITest (236080 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:47:01 dist-test-slave-j5c3 krb5kdc[18881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:47:01 dist-test-slave-j5c3 krb5kdc[18881](info): set up 2 sockets
Sep 28 04:47:01 dist-test-slave-j5c3 krb5kdc[18881](info): commencing operation
krb5kdc: starting...
W20260928 04:47:02.847934 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:47:02 dist-test-slave-j5c3 krb5kdc[18881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:47:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:47:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:47:02.975698 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:41465
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:41665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:41465
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:03.060451 18894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:03.060699 18894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:03.060736 18894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:03.061903 18894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:03.061975 18894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:03.061998 18894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:03.062031 18894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:03.062057 18894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:03.063560 18894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:41665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:41465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:41465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:03.063855 18894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:03.064098 18894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:03.069209 18902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:03.069288 18899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:03.069523 18900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:03.069518 18894 server_base.cc:1034] running on GCE node
I20260928 04:47:03.069923 18894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:03.070209 18894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:03.071383 18894 hybrid_clock.cc:648] HybridClock initialized: now 1790570823071351 us; error 50 us; skew 500 ppm
Sep 28 04:47:03 dist-test-slave-j5c3 krb5kdc[18881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570823, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:03.074262 18894 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:47:03.074790 18894 webserver.cc:460] Webserver started at http://127.1.226.254:45497/ using document root <none> and password file <none>
I20260928 04:47:03.074993 18894 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:03.075038 18894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:03.075137 18894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:03.076323 18894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "498453cced3043a9a7250b7b72382a94"
format_stamp: "Formatted at 2026-09-28 04:47:03 on dist-test-slave-j5c3"
server_key: "67f91de8543a02509256b20115f06dfa"
server_key_iv: "e063a74bf2d7129cb11b1f8d6468f129"
server_key_version: "encryptionkey@0"
I20260928 04:47:03.076674 18894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "498453cced3043a9a7250b7b72382a94"
format_stamp: "Formatted at 2026-09-28 04:47:03 on dist-test-slave-j5c3"
server_key: "67f91de8543a02509256b20115f06dfa"
server_key_iv: "e063a74bf2d7129cb11b1f8d6468f129"
server_key_version: "encryptionkey@0"
I20260928 04:47:03.078122 18894 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:47:03.078961 18909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:03.079267 18894 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:03.079401 18894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "498453cced3043a9a7250b7b72382a94"
format_stamp: "Formatted at 2026-09-28 04:47:03 on dist-test-slave-j5c3"
server_key: "67f91de8543a02509256b20115f06dfa"
server_key_iv: "e063a74bf2d7129cb11b1f8d6468f129"
server_key_version: "encryptionkey@0"
I20260928 04:47:03.079487 18894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:03.095793 18894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:03.097996 18894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:03.098186 18894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:03.102831 18894 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:41465
I20260928 04:47:03.102895 18961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:41465 every 8 connection(s)
I20260928 04:47:03.103276 18894 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:47:03.104264 18962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:03.106873 18962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 498453cced3043a9a7250b7b72382a94: Bootstrap starting.
I20260928 04:47:03.107637 18962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 498453cced3043a9a7250b7b72382a94: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:03.107988 18962 log.cc:824] T 00000000000000000000000000000000 P 498453cced3043a9a7250b7b72382a94: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:03.108837 18962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 498453cced3043a9a7250b7b72382a94: No bootstrap required, opened a new log
I20260928 04:47:03.110713 18962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 498453cced3043a9a7250b7b72382a94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498453cced3043a9a7250b7b72382a94" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 41465 } }
I20260928 04:47:03.110881 18962 raft_consensus.cc:374] T 00000000000000000000000000000000 P 498453cced3043a9a7250b7b72382a94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:03.110908 18962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 498453cced3043a9a7250b7b72382a94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 498453cced3043a9a7250b7b72382a94, State: Initialized, Role: FOLLOWER
I20260928 04:47:03.111073 18962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 498453cced3043a9a7250b7b72382a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498453cced3043a9a7250b7b72382a94" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 41465 } }
I20260928 04:47:03.111192 18962 raft_consensus.cc:388] T 00000000000000000000000000000000 P 498453cced3043a9a7250b7b72382a94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:47:03.111258 18962 raft_consensus.cc:482] T 00000000000000000000000000000000 P 498453cced3043a9a7250b7b72382a94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:47:03.111330 18962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 498453cced3043a9a7250b7b72382a94 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:03.111402 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 18894
I20260928 04:47:03.111541 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:47:03.111828 1931 external_mini_cluster.cc:1428] Setting key 4dd337c27e10287ab87c982b3fda47d0
I20260928 04:47:03.112186 18962 raft_consensus.cc:504] T 00000000000000000000000000000000 P 498453cced3043a9a7250b7b72382a94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498453cced3043a9a7250b7b72382a94" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 41465 } }
I20260928 04:47:03.112350 18962 leader_election.cc:302] T 00000000000000000000000000000000 P 498453cced3043a9a7250b7b72382a94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 498453cced3043a9a7250b7b72382a94; no voters:
I20260928 04:47:03.112605 18962 leader_election.cc:288] T 00000000000000000000000000000000 P 498453cced3043a9a7250b7b72382a94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:47:03.112643 18966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 498453cced3043a9a7250b7b72382a94 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:03.112759 18966 raft_consensus.cc:686] T 00000000000000000000000000000000 P 498453cced3043a9a7250b7b72382a94 [term 1 LEADER]: Becoming Leader. State: Replica: 498453cced3043a9a7250b7b72382a94, State: Running, Role: LEADER
I20260928 04:47:03.112917 18962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 498453cced3043a9a7250b7b72382a94 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:03.112973 18966 consensus_queue.cc:237] T 00000000000000000000000000000000 P 498453cced3043a9a7250b7b72382a94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498453cced3043a9a7250b7b72382a94" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 41465 } }
I20260928 04:47:03.113692 18968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 498453cced3043a9a7250b7b72382a94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "498453cced3043a9a7250b7b72382a94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498453cced3043a9a7250b7b72382a94" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 41465 } } }
I20260928 04:47:03.114207 18968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 498453cced3043a9a7250b7b72382a94 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:03.114091 18969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 498453cced3043a9a7250b7b72382a94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 498453cced3043a9a7250b7b72382a94. Latest consensus state: current_term: 1 leader_uuid: "498453cced3043a9a7250b7b72382a94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498453cced3043a9a7250b7b72382a94" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 41465 } } }
I20260928 04:47:03.114494 18982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:47:03.115235 18982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:47:03.114542 18969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 498453cced3043a9a7250b7b72382a94 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:03.117050 18982 catalog_manager.cc:1347] Generated new cluster ID: f79cb481bb0e43b8be3d29d11bb444c1
I20260928 04:47:03.117097 18982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:47:03 dist-test-slave-j5c3 krb5kdc[18881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:47:03.125088 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:03.112543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43512 (local address 127.1.226.254:41465)
0928 04:47:03.112875 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:03.112887 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:03.112929 (+ 42us) server_negotiation.cc:407] Connection header received
0928 04:47:03.113076 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:03.113083 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:03.113138 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:03.113309 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:03.114209 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.115672 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.116765 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.117397 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.117498 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:03.120423 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:03.120427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:03.120427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:03.120441 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:03.122940 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:03.123367 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:03.123370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:03.123370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:03.123492 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:03.123712 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:03.123713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:03.123713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:03.124014 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:03.124180 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:03.124690 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 04:47:03.124853 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":281,"thread_start_us":106,"threads_started":1}
I20260928 04:47:03.127283 18982 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:47:03.127774 18982 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:47:03.132491 18982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 498453cced3043a9a7250b7b72382a94: Generated new TSK 0
I20260928 04:47:03.132709 18982 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.226.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
I20260928 04:47:03.165480 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:0
--local_ip_for_outbound_sockets=127.1.226.193
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:41465
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:41665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:03.257128 18988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:03.257349 18988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:03.257372 18988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:03.258534 18988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:03.258575 18988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:03.258601 18988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:47:03.260066 18988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:41665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:41465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:03.260353 18988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:03.260592 18988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:03.265885 18993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:03.265873 18996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:03.266060 18988 server_base.cc:1034] running on GCE node
W20260928 04:47:03.266155 18994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:03.266403 18988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:03.266674 18988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:03.267858 18988 hybrid_clock.cc:648] HybridClock initialized: now 1790570823267828 us; error 48 us; skew 500 ppm
Sep 28 04:47:03 dist-test-slave-j5c3 krb5kdc[18881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570823, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:03.270704 18988 init.cc:377] Logged in from keytab as kudu/127.1.226.193@KRBTEST.COM (short username kudu)
I20260928 04:47:03.271248 18988 webserver.cc:460] Webserver started at http://127.1.226.193:33701/ using document root <none> and password file <none>
I20260928 04:47:03.271497 18988 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:03.271566 18988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:03.271726 18988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:03.272914 18988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-0/data/instance:
uuid: "d578ba74e64a416ab5e9b621d7e56679"
format_stamp: "Formatted at 2026-09-28 04:47:03 on dist-test-slave-j5c3"
server_key: "abbc4c33358aa3109b8eb1079153a8e9"
server_key_iv: "4b43c9147e56663280da964dd3fc7320"
server_key_version: "encryptionkey@0"
I20260928 04:47:03.273289 18988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance:
uuid: "d578ba74e64a416ab5e9b621d7e56679"
format_stamp: "Formatted at 2026-09-28 04:47:03 on dist-test-slave-j5c3"
server_key: "abbc4c33358aa3109b8eb1079153a8e9"
server_key_iv: "4b43c9147e56663280da964dd3fc7320"
server_key_version: "encryptionkey@0"
I20260928 04:47:03.274634 18988 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:47:03.275390 19003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:03.275566 18988 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:47:03.275673 18988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "d578ba74e64a416ab5e9b621d7e56679"
format_stamp: "Formatted at 2026-09-28 04:47:03 on dist-test-slave-j5c3"
server_key: "abbc4c33358aa3109b8eb1079153a8e9"
server_key_iv: "4b43c9147e56663280da964dd3fc7320"
server_key_version: "encryptionkey@0"
I20260928 04:47:03.275749 18988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:03.300263 18988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:03.302523 18988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:03.302762 18988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:03.303037 18988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:47:03.303412 18988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:47:03.303471 18988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:03.303512 18988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:47:03.303540 18988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:03.309459 18988 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:42427
I20260928 04:47:03.309510 19116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:42427 every 8 connection(s)
I20260928 04:47:03.309872 18988 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
I20260928 04:47:03.311096 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 18988
I20260928 04:47:03.311188 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance
I20260928 04:47:03.311393 1931 external_mini_cluster.cc:1428] Setting key 819666191fa0893ab1a49b2dbb7982c3
Sep 28 04:47:03 dist-test-slave-j5c3 krb5kdc[18881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570823, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.226.194@KRBTEST.COM; defaulting to no policy
I20260928 04:47:03.320767 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:03.310590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:58597 (local address 127.1.226.254:41465)
0928 04:47:03.310637 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:03.310640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:03.310855 (+ 215us) server_negotiation.cc:407] Connection header received
0928 04:47:03.311462 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:03.311464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:03.311478 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:03.311545 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:03.312654 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.313490 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.315073 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.315295 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.315376 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:03.317707 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:03.317709 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:03.317709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:03.317718 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:03.319378 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:03.319792 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:03.319793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:03.319794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:03.319865 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:03.320112 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:03.320113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:03.320113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:03.320293 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:03.320366 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:03.320627 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 04:47:03.320698 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:47:03.320984 19119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:03.310555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:41465 (local address 127.1.226.193:58597)
0928 04:47:03.310725 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:03.310762 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:03.311345 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:03.311643 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:03.311648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:03.312354 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:03.312555 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:03.312560 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.313612 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.313613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:03.314954 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:03.314957 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.315384 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.315385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:03.315532 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:03.316142 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:47:03.316194 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:03.317607 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:03.319473 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:03.319480 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:03.319482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:03.319712 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:03.319938 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:03.319938 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:03.319939 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:03.320041 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:03.320373 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:03.320379 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:03.320501 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 04:47:03.320591 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 04:47:03.320717 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":100,"threads_started":1}
I20260928 04:47:03.321550 19117 heartbeater.cc:347] Connected to a master server at 127.1.226.254:41465
I20260928 04:47:03.321673 19117 heartbeater.cc:464] Registering TS with master...
I20260928 04:47:03.321861 19117 heartbeater.cc:511] Master 127.1.226.254:41465 requested a full tablet report, sending...
I20260928 04:47:03.322504 18926 ts_manager.cc:194] Registered new tserver with Master: d578ba74e64a416ab5e9b621d7e56679 (127.1.226.193:42427)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.194@KRBTEST.COM" created.
I20260928 04:47:03.323377 18926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.193@KRBTEST.COM'} at 127.1.226.193:58597
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
I20260928 04:47:03.337549 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:0
--local_ip_for_outbound_sockets=127.1.226.194
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:41465
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:41665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:03.421237 19122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:03.421458 19122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:03.421482 19122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:03.422678 19122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:03.422719 19122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:03.422753 19122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:47:03.424180 19122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:41665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:41465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:03.424376 19122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:03.424553 19122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:03.430030 19128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:03.430043 19122 server_base.cc:1034] running on GCE node
W20260928 04:47:03.430009 19127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:03.430009 19130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:03.430464 19122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:03.430682 19122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:03.431839 19122 hybrid_clock.cc:648] HybridClock initialized: now 1790570823431818 us; error 36 us; skew 500 ppm
Sep 28 04:47:03 dist-test-slave-j5c3 krb5kdc[18881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570823, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:03.434417 19122 init.cc:377] Logged in from keytab as kudu/127.1.226.194@KRBTEST.COM (short username kudu)
I20260928 04:47:03.434841 19122 webserver.cc:460] Webserver started at http://127.1.226.194:46329/ using document root <none> and password file <none>
I20260928 04:47:03.435034 19122 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:03.435074 19122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:03.435163 19122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:03.436198 19122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-1/data/instance:
uuid: "ec6180d606ad4c85bf8556123c6d5002"
format_stamp: "Formatted at 2026-09-28 04:47:03 on dist-test-slave-j5c3"
server_key: "bd5254db089b0d0caf7ab36773049413"
server_key_iv: "47d9f4b65e982a356e461cefc1bd3648"
server_key_version: "encryptionkey@0"
I20260928 04:47:03.436532 19122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance:
uuid: "ec6180d606ad4c85bf8556123c6d5002"
format_stamp: "Formatted at 2026-09-28 04:47:03 on dist-test-slave-j5c3"
server_key: "bd5254db089b0d0caf7ab36773049413"
server_key_iv: "47d9f4b65e982a356e461cefc1bd3648"
server_key_version: "encryptionkey@0"
I20260928 04:47:03.437846 19122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:03.438539 19137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:03.438709 19122 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:47:03.438799 19122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "ec6180d606ad4c85bf8556123c6d5002"
format_stamp: "Formatted at 2026-09-28 04:47:03 on dist-test-slave-j5c3"
server_key: "bd5254db089b0d0caf7ab36773049413"
server_key_iv: "47d9f4b65e982a356e461cefc1bd3648"
server_key_version: "encryptionkey@0"
I20260928 04:47:03.438855 19122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:03.453961 19122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:03.456264 19122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:03.456384 19122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:03.456584 19122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:47:03.456924 19122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:47:03.456965 19122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:03.456988 19122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:47:03.457000 19122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:03.462618 19122 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:40481
I20260928 04:47:03.462705 19250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:40481 every 8 connection(s)
I20260928 04:47:03.463034 19122 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
Sep 28 04:47:03 dist-test-slave-j5c3 krb5kdc[18881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570823, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:47:03.472631 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:03.463911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:51731 (local address 127.1.226.254:41465)
0928 04:47:03.463997 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:03.463999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:03.464199 (+ 200us) server_negotiation.cc:407] Connection header received
0928 04:47:03.464756 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:03.464757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:03.464768 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:03.464828 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:03.465800 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.466288 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.467382 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.467563 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.467613 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:03.469991 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:03.469992 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:03.469992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:03.469999 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:03.471221 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:03.471635 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:03.471636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:03.471636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:03.471689 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:03.471916 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:03.471917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:03.471917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:03.472021 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:03.472074 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:03.472492 (+ 418us) server_negotiation.cc:299] Negotiation successful
0928 04:47:03.472563 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 04:47:03.472908 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:03.463872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:41465 (local address 127.1.226.194:51731)
0928 04:47:03.464072 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:03.464112 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:03.464663 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:03.464889 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:03.464893 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:03.465505 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:03.465709 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:03.465714 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.466385 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.466386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:03.467300 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:03.467302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.467635 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.467635 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:03.467742 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:03.468445 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:47:03.468453 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:03.469892 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:03.471302 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:03.471308 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:03.471310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:03.471567 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:03.471753 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:03.471754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:03.471754 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:03.471859 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:03.472108 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:03.472114 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:03.472348 (+ 234us) client_negotiation.cc:769] Sending connection context
0928 04:47:03.472469 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 04:47:03.472608 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":106,"threads_started":1}
I20260928 04:47:03.473052 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 19122
I20260928 04:47:03.473215 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance
I20260928 04:47:03.473477 1931 external_mini_cluster.cc:1428] Setting key 97787ef122b127268550994d592ebe39
I20260928 04:47:03.473505 19251 heartbeater.cc:347] Connected to a master server at 127.1.226.254:41465
I20260928 04:47:03.473616 19251 heartbeater.cc:464] Registering TS with master...
I20260928 04:47:03.473835 19251 heartbeater.cc:511] Master 127.1.226.254:41465 requested a full tablet report, sending...
I20260928 04:47:03.474282 18926 ts_manager.cc:194] Registered new tserver with Master: ec6180d606ad4c85bf8556123c6d5002 (127.1.226.194:40481)
I20260928 04:47:03.474649 18926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.194@KRBTEST.COM'} at 127.1.226.194:51731
WARNING: no policy specified for kudu/127.1.226.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
I20260928 04:47:03.501075 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:0
--local_ip_for_outbound_sockets=127.1.226.195
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:41465
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:41665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:03.589911 19256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:03.590160 19256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:03.590199 19256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:03.591384 19256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:03.591451 19256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:03.591495 19256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:47:03.593046 19256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:41665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:41465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:03.593329 19256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:03.593566 19256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:03.599376 19261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:03.599376 19264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:03.599642 19256 server_base.cc:1034] running on GCE node
W20260928 04:47:03.599401 19262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:03.599900 19256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:03.600199 19256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:03.601366 19256 hybrid_clock.cc:648] HybridClock initialized: now 1790570823601360 us; error 38 us; skew 500 ppm
Sep 28 04:47:03 dist-test-slave-j5c3 krb5kdc[18881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570823, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:03.604295 19256 init.cc:377] Logged in from keytab as kudu/127.1.226.195@KRBTEST.COM (short username kudu)
I20260928 04:47:03.604888 19256 webserver.cc:460] Webserver started at http://127.1.226.195:46035/ using document root <none> and password file <none>
I20260928 04:47:03.605150 19256 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:03.605226 19256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:03.605396 19256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:03.606518 19256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-2/data/instance:
uuid: "8655ad16feb84569ac8cdfc7a908a6b3"
format_stamp: "Formatted at 2026-09-28 04:47:03 on dist-test-slave-j5c3"
server_key: "a1bb56b55c4e82c733252d1d2953bdd4"
server_key_iv: "8ba98c449873f99f11a947ca38b0d5d3"
server_key_version: "encryptionkey@0"
I20260928 04:47:03.606940 19256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance:
uuid: "8655ad16feb84569ac8cdfc7a908a6b3"
format_stamp: "Formatted at 2026-09-28 04:47:03 on dist-test-slave-j5c3"
server_key: "a1bb56b55c4e82c733252d1d2953bdd4"
server_key_iv: "8ba98c449873f99f11a947ca38b0d5d3"
server_key_version: "encryptionkey@0"
I20260928 04:47:03.608369 19256 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:47:03.609295 19271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:03.609505 19256 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:03.609618 19256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "8655ad16feb84569ac8cdfc7a908a6b3"
format_stamp: "Formatted at 2026-09-28 04:47:03 on dist-test-slave-j5c3"
server_key: "a1bb56b55c4e82c733252d1d2953bdd4"
server_key_iv: "8ba98c449873f99f11a947ca38b0d5d3"
server_key_version: "encryptionkey@0"
I20260928 04:47:03.609767 19256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:03.619783 19256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:03.622517 19256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:03.622695 19256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:03.623071 19256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:47:03.623512 19256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:47:03.623634 19256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:03.623698 19256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:47:03.625722 19256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 04:47:03.633872 19256 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:41423
I20260928 04:47:03.633916 19384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:41423 every 8 connection(s)
I20260928 04:47:03.634291 19256 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
I20260928 04:47:03.636354 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 19256
I20260928 04:47:03.636447 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance
I20260928 04:47:03.636622 1931 external_mini_cluster.cc:1428] Setting key 8b917c9f7664a8ed190f0737037997fe
Sep 28 04:47:03 dist-test-slave-j5c3 krb5kdc[18881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570823, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:47:03.644335 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:03.635050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:34261 (local address 127.1.226.254:41465)
0928 04:47:03.635131 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:03.635134 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:03.635446 (+ 312us) server_negotiation.cc:407] Connection header received
0928 04:47:03.635896 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:03.635898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:03.635909 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:03.635977 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:03.637345 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.637951 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.638986 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.639184 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.639257 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:03.641560 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:03.641561 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:03.641562 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:03.641577 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:03.642971 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:03.643367 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:03.643368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:03.643368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:03.643427 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:03.643654 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:03.643654 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:03.643654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:03.643761 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:03.643812 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:03.644139 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 04:47:03.644249 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 04:47:03.644814 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:03.635045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:41465 (local address 127.1.226.195:34261)
0928 04:47:03.635227 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:03.635273 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:03.635794 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:03.636034 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:03.636038 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:03.636949 (+ 911us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:03.637264 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:03.637271 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.638058 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.638060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:03.638883 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:03.638884 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.639274 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.639275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:03.639387 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:03.639963 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:47:03.639972 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:03.641439 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:03.643054 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:03.643061 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:03.643063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:03.643297 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:03.643488 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:03.643489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:03.643489 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:03.643599 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:03.643844 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:03.643851 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:03.644006 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 04:47:03.644113 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 04:47:03.644481 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260928 04:47:03.645604 19385 heartbeater.cc:347] Connected to a master server at 127.1.226.254:41465
I20260928 04:47:03.645798 19385 heartbeater.cc:464] Registering TS with master...
I20260928 04:47:03.646021 19385 heartbeater.cc:511] Master 127.1.226.254:41465 requested a full tablet report, sending...
I20260928 04:47:03.646428 18926 ts_manager.cc:194] Registered new tserver with Master: 8655ad16feb84569ac8cdfc7a908a6b3 (127.1.226.195:41423)
I20260928 04:47:03.646776 18926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.195@KRBTEST.COM'} at 127.1.226.195:34261
I20260928 04:47:03.648528 1931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:47:03.656245 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:03.649953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43524 (local address 127.1.226.254:41465)
0928 04:47:03.650005 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:03.650007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:03.650183 (+ 176us) server_negotiation.cc:407] Connection header received
0928 04:47:03.650253 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:03.650254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:03.650263 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:03.650309 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:03.650919 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.651399 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.652135 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.652364 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.652450 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:03.653607 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:03.653609 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:03.653609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:03.653617 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:03.654898 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:03.655345 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:03.655347 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:03.655347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:03.655410 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:03.655615 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:03.655616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:03.655617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:03.655763 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:03.655868 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:03.656047 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 04:47:03.656124 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 04:47:03.657756 18926 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:47:03.659250 18926 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:47:03.665655 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:03.661683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:42427 (local address 127.0.0.1:50004)
0928 04:47:03.662182 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:03.662190 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:03.662236 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:03.662511 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:03.662512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:03.662514 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:03.662545 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:03.662547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.663798 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.663799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:03.664628 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:03.664630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.665352 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.665353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:03.665432 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:03.665446 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 04:47:03.665525 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 04:47:03.665577 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":99,"threads_started":1}
I20260928 04:47:03.665678 19401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:03.661971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50004 (local address 127.1.226.193:42427)
0928 04:47:03.662154 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:03.662159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:03.662304 (+ 145us) server_negotiation.cc:407] Connection header received
0928 04:47:03.662349 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:03.662351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:03.662366 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:03.662435 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:03.662618 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.663688 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.664727 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.665268 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.665343 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:03.665401 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:03.665554 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 04:47:03.665595 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":78,"threads_started":1}
I20260928 04:47:03.666064 19400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:03.661846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33694 (local address 127.1.226.195:41423)
0928 04:47:03.662097 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:03.662103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:03.662231 (+ 128us) server_negotiation.cc:407] Connection header received
0928 04:47:03.662358 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:03.662361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:03.662378 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:03.662434 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:03.662649 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.663329 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.664775 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.665482 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.665542 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:03.665593 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:03.665917 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 04:47:03.665980 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":135,"threads_started":1}
I20260928 04:47:03.666393 19396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:03.661668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:41423 (local address 127.0.0.1:33694)
0928 04:47:03.662144 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:03.662155 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:03.662263 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:03.662537 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:03.662538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:03.662539 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:03.662594 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:03.662596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.663447 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.663449 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:03.664687 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:03.664690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.665624 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.665625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:03.665686 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:03.665689 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:47:03.665740 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 04:47:03.666329 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20260928 04:47:03.666528 19399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:03.661868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40906 (local address 127.1.226.194:40481)
0928 04:47:03.662069 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:03.662074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:03.662085 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:47:03.662155 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:03.662157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:03.662175 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:03.662261 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:03.662510 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.663660 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.665179 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.666054 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.666143 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:03.666195 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:03.666409 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 04:47:03.666450 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":85,"threads_started":1}
I20260928 04:47:03.666517 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:03.661668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:40481 (local address 127.0.0.1:40906)
0928 04:47:03.661961 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:03.662005 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:03.662074 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:03.662348 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:03.662351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:03.662353 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:03.662432 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:03.662434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.663768 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.663769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:03.665096 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:03.665098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.666146 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.666147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:03.666334 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:03.666338 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:47:03.666394 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 04:47:03.666446 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":110,"thread_start_us":67,"threads_started":1}
I20260928 04:47:03.666654 19051 tablet_service.cc:1467] Processing CreateTablet for tablet 682fa61fdc3b41a8b3b16e8606a11725 (DEFAULT_TABLE table=test-table [id=98044a5c320a40daa06fbbcb2a0726f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:47:03.667037 19051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 682fa61fdc3b41a8b3b16e8606a11725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:03.667253 19319 tablet_service.cc:1467] Processing CreateTablet for tablet 682fa61fdc3b41a8b3b16e8606a11725 (DEFAULT_TABLE table=test-table [id=98044a5c320a40daa06fbbcb2a0726f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:47:03.667482 19319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 682fa61fdc3b41a8b3b16e8606a11725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:03.667418 19185 tablet_service.cc:1467] Processing CreateTablet for tablet 682fa61fdc3b41a8b3b16e8606a11725 (DEFAULT_TABLE table=test-table [id=98044a5c320a40daa06fbbcb2a0726f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:47:03.667716 19185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 682fa61fdc3b41a8b3b16e8606a11725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:03.669924 19402 tablet_bootstrap.cc:492] T 682fa61fdc3b41a8b3b16e8606a11725 P 8655ad16feb84569ac8cdfc7a908a6b3: Bootstrap starting.
I20260928 04:47:03.670408 19403 tablet_bootstrap.cc:492] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679: Bootstrap starting.
I20260928 04:47:03.670595 19402 tablet_bootstrap.cc:654] T 682fa61fdc3b41a8b3b16e8606a11725 P 8655ad16feb84569ac8cdfc7a908a6b3: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:03.671135 19404 tablet_bootstrap.cc:492] T 682fa61fdc3b41a8b3b16e8606a11725 P ec6180d606ad4c85bf8556123c6d5002: Bootstrap starting.
I20260928 04:47:03.671164 19403 tablet_bootstrap.cc:654] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:03.671227 19402 log.cc:824] T 682fa61fdc3b41a8b3b16e8606a11725 P 8655ad16feb84569ac8cdfc7a908a6b3: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:03.671687 19403 log.cc:824] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:03.671773 19404 tablet_bootstrap.cc:654] T 682fa61fdc3b41a8b3b16e8606a11725 P ec6180d606ad4c85bf8556123c6d5002: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:03.672091 19402 tablet_bootstrap.cc:492] T 682fa61fdc3b41a8b3b16e8606a11725 P 8655ad16feb84569ac8cdfc7a908a6b3: No bootstrap required, opened a new log
I20260928 04:47:03.672212 19402 ts_tablet_manager.cc:1397] T 682fa61fdc3b41a8b3b16e8606a11725 P 8655ad16feb84569ac8cdfc7a908a6b3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:47:03.672413 19404 log.cc:824] T 682fa61fdc3b41a8b3b16e8606a11725 P ec6180d606ad4c85bf8556123c6d5002: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:03.672528 19403 tablet_bootstrap.cc:492] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679: No bootstrap required, opened a new log
I20260928 04:47:03.672596 19403 ts_tablet_manager.cc:1397] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:47:03.674139 19402 raft_consensus.cc:348] T 682fa61fdc3b41a8b3b16e8606a11725 P 8655ad16feb84569ac8cdfc7a908a6b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6180d606ad4c85bf8556123c6d5002" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40481 } } peers { permanent_uuid: "8655ad16feb84569ac8cdfc7a908a6b3" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 41423 } } peers { permanent_uuid: "d578ba74e64a416ab5e9b621d7e56679" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 42427 } }
I20260928 04:47:03.674341 19402 raft_consensus.cc:374] T 682fa61fdc3b41a8b3b16e8606a11725 P 8655ad16feb84569ac8cdfc7a908a6b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:03.674377 19402 raft_consensus.cc:729] T 682fa61fdc3b41a8b3b16e8606a11725 P 8655ad16feb84569ac8cdfc7a908a6b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8655ad16feb84569ac8cdfc7a908a6b3, State: Initialized, Role: FOLLOWER
I20260928 04:47:03.674487 19402 consensus_queue.cc:260] T 682fa61fdc3b41a8b3b16e8606a11725 P 8655ad16feb84569ac8cdfc7a908a6b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6180d606ad4c85bf8556123c6d5002" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40481 } } peers { permanent_uuid: "8655ad16feb84569ac8cdfc7a908a6b3" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 41423 } } peers { permanent_uuid: "d578ba74e64a416ab5e9b621d7e56679" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 42427 } }
I20260928 04:47:03.674515 19403 raft_consensus.cc:348] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6180d606ad4c85bf8556123c6d5002" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40481 } } peers { permanent_uuid: "8655ad16feb84569ac8cdfc7a908a6b3" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 41423 } } peers { permanent_uuid: "d578ba74e64a416ab5e9b621d7e56679" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 42427 } }
I20260928 04:47:03.674685 19403 raft_consensus.cc:374] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:03.674743 19403 raft_consensus.cc:729] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d578ba74e64a416ab5e9b621d7e56679, State: Initialized, Role: FOLLOWER
I20260928 04:47:03.674772 19402 ts_tablet_manager.cc:1428] T 682fa61fdc3b41a8b3b16e8606a11725 P 8655ad16feb84569ac8cdfc7a908a6b3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 04:47:03.674813 19385 heartbeater.cc:503] Master 127.1.226.254:41465 was elected leader, sending a full tablet report...
I20260928 04:47:03.674858 19403 consensus_queue.cc:260] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6180d606ad4c85bf8556123c6d5002" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40481 } } peers { permanent_uuid: "8655ad16feb84569ac8cdfc7a908a6b3" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 41423 } } peers { permanent_uuid: "d578ba74e64a416ab5e9b621d7e56679" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 42427 } }
I20260928 04:47:03.675139 19403 ts_tablet_manager.cc:1428] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 04:47:03.675292 19117 heartbeater.cc:503] Master 127.1.226.254:41465 was elected leader, sending a full tablet report...
I20260928 04:47:03.675370 19404 tablet_bootstrap.cc:492] T 682fa61fdc3b41a8b3b16e8606a11725 P ec6180d606ad4c85bf8556123c6d5002: No bootstrap required, opened a new log
I20260928 04:47:03.675426 19404 ts_tablet_manager.cc:1397] T 682fa61fdc3b41a8b3b16e8606a11725 P ec6180d606ad4c85bf8556123c6d5002: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260928 04:47:03.677026 19404 raft_consensus.cc:348] T 682fa61fdc3b41a8b3b16e8606a11725 P ec6180d606ad4c85bf8556123c6d5002 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6180d606ad4c85bf8556123c6d5002" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40481 } } peers { permanent_uuid: "8655ad16feb84569ac8cdfc7a908a6b3" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 41423 } } peers { permanent_uuid: "d578ba74e64a416ab5e9b621d7e56679" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 42427 } }
I20260928 04:47:03.677162 19404 raft_consensus.cc:374] T 682fa61fdc3b41a8b3b16e8606a11725 P ec6180d606ad4c85bf8556123c6d5002 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:03.677184 19404 raft_consensus.cc:729] T 682fa61fdc3b41a8b3b16e8606a11725 P ec6180d606ad4c85bf8556123c6d5002 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec6180d606ad4c85bf8556123c6d5002, State: Initialized, Role: FOLLOWER
I20260928 04:47:03.677261 19404 consensus_queue.cc:260] T 682fa61fdc3b41a8b3b16e8606a11725 P ec6180d606ad4c85bf8556123c6d5002 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6180d606ad4c85bf8556123c6d5002" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40481 } } peers { permanent_uuid: "8655ad16feb84569ac8cdfc7a908a6b3" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 41423 } } peers { permanent_uuid: "d578ba74e64a416ab5e9b621d7e56679" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 42427 } }
I20260928 04:47:03.677451 19404 ts_tablet_manager.cc:1428] T 682fa61fdc3b41a8b3b16e8606a11725 P ec6180d606ad4c85bf8556123c6d5002: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 04:47:03.677529 19251 heartbeater.cc:503] Master 127.1.226.254:41465 was elected leader, sending a full tablet report...
W20260928 04:47:03.713799 19252 tablet.cc:2369] T 682fa61fdc3b41a8b3b16e8606a11725 P ec6180d606ad4c85bf8556123c6d5002: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:47:03.810650 19118 tablet.cc:2369] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:47:03.825609 19409 raft_consensus.cc:482] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:47:03.825733 19409 raft_consensus.cc:504] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6180d606ad4c85bf8556123c6d5002" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40481 } } peers { permanent_uuid: "8655ad16feb84569ac8cdfc7a908a6b3" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 41423 } } peers { permanent_uuid: "d578ba74e64a416ab5e9b621d7e56679" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 42427 } }
I20260928 04:47:03.826045 19409 leader_election.cc:288] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec6180d606ad4c85bf8556123c6d5002 (127.1.226.194:40481), 8655ad16feb84569ac8cdfc7a908a6b3 (127.1.226.195:41423)
I20260928 04:47:03.830191 19399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:03.826392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:55039 (local address 127.1.226.194:40481)
0928 04:47:03.826452 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:03.826455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:03.826540 (+ 85us) server_negotiation.cc:407] Connection header received
0928 04:47:03.826598 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:03.826599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:03.826609 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:03.826663 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:03.826868 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.827624 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.829179 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.829781 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.829900 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:03.829908 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:03.830079 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 04:47:03.830118 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 04:47:03.830191 19400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:03.826371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:54241 (local address 127.1.226.195:41423)
0928 04:47:03.826427 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:03.826429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:03.826469 (+ 40us) server_negotiation.cc:407] Connection header received
0928 04:47:03.826583 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:03.826585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:03.826595 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:03.826651 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:03.826868 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.827624 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.829179 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.829781 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.829864 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:03.829876 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:03.830079 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 04:47:03.830118 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 04:47:03.830235 19411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:03.826278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:40481 (local address 127.1.226.193:55039)
0928 04:47:03.826445 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:03.826457 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:03.826539 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:03.826679 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:03.826679 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:03.826680 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:03.826762 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:03.826764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.827712 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.827712 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:03.829081 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:03.829084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.829907 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.829908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:03.829995 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:03.830003 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 04:47:03.830063 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 04:47:03.830118 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":100,"thread_start_us":60,"threads_started":1}
I20260928 04:47:03.830237 19412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:03.826259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:41423 (local address 127.1.226.193:54241)
0928 04:47:03.826404 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:03.826418 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:03.826488 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:03.826676 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:03.826677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:03.826679 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:03.826767 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:03.826768 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.827711 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.827712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:03.829081 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:03.829084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.829908 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.829909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:03.829995 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:03.830002 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 04:47:03.830063 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 04:47:03.830118 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":34,"thread_start_us":99,"threads_started":1}
I20260928 04:47:03.830628 19205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682fa61fdc3b41a8b3b16e8606a11725" candidate_uuid: "d578ba74e64a416ab5e9b621d7e56679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec6180d606ad4c85bf8556123c6d5002" is_pre_election: true
I20260928 04:47:03.830673 19339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682fa61fdc3b41a8b3b16e8606a11725" candidate_uuid: "d578ba74e64a416ab5e9b621d7e56679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8655ad16feb84569ac8cdfc7a908a6b3" is_pre_election: true
I20260928 04:47:03.830787 19205 raft_consensus.cc:2454] T 682fa61fdc3b41a8b3b16e8606a11725 P ec6180d606ad4c85bf8556123c6d5002 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d578ba74e64a416ab5e9b621d7e56679 in term 0.
I20260928 04:47:03.830837 19339 raft_consensus.cc:2454] T 682fa61fdc3b41a8b3b16e8606a11725 P 8655ad16feb84569ac8cdfc7a908a6b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d578ba74e64a416ab5e9b621d7e56679 in term 0.
I20260928 04:47:03.831020 19004 leader_election.cc:302] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d578ba74e64a416ab5e9b621d7e56679, ec6180d606ad4c85bf8556123c6d5002; no voters:
I20260928 04:47:03.831207 19409 raft_consensus.cc:2781] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:47:03.831248 19409 raft_consensus.cc:482] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:47:03.831264 19409 raft_consensus.cc:3037] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:03.831933 19409 raft_consensus.cc:504] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6180d606ad4c85bf8556123c6d5002" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40481 } } peers { permanent_uuid: "8655ad16feb84569ac8cdfc7a908a6b3" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 41423 } } peers { permanent_uuid: "d578ba74e64a416ab5e9b621d7e56679" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 42427 } }
I20260928 04:47:03.832075 19409 leader_election.cc:288] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679 [CANDIDATE]: Term 1 election: Requested vote from peers ec6180d606ad4c85bf8556123c6d5002 (127.1.226.194:40481), 8655ad16feb84569ac8cdfc7a908a6b3 (127.1.226.195:41423)
I20260928 04:47:03.832248 19205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682fa61fdc3b41a8b3b16e8606a11725" candidate_uuid: "d578ba74e64a416ab5e9b621d7e56679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec6180d606ad4c85bf8556123c6d5002"
I20260928 04:47:03.832247 19339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682fa61fdc3b41a8b3b16e8606a11725" candidate_uuid: "d578ba74e64a416ab5e9b621d7e56679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8655ad16feb84569ac8cdfc7a908a6b3"
I20260928 04:47:03.832336 19205 raft_consensus.cc:3037] T 682fa61fdc3b41a8b3b16e8606a11725 P ec6180d606ad4c85bf8556123c6d5002 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:03.832341 19339 raft_consensus.cc:3037] T 682fa61fdc3b41a8b3b16e8606a11725 P 8655ad16feb84569ac8cdfc7a908a6b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:03.833060 19205 raft_consensus.cc:2454] T 682fa61fdc3b41a8b3b16e8606a11725 P ec6180d606ad4c85bf8556123c6d5002 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d578ba74e64a416ab5e9b621d7e56679 in term 1.
I20260928 04:47:03.833146 19339 raft_consensus.cc:2454] T 682fa61fdc3b41a8b3b16e8606a11725 P 8655ad16feb84569ac8cdfc7a908a6b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d578ba74e64a416ab5e9b621d7e56679 in term 1.
I20260928 04:47:03.833220 19004 leader_election.cc:302] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d578ba74e64a416ab5e9b621d7e56679, ec6180d606ad4c85bf8556123c6d5002; no voters:
I20260928 04:47:03.833307 19409 raft_consensus.cc:2781] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:03.833460 19409 raft_consensus.cc:686] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679 [term 1 LEADER]: Becoming Leader. State: Replica: d578ba74e64a416ab5e9b621d7e56679, State: Running, Role: LEADER
I20260928 04:47:03.833657 19409 consensus_queue.cc:237] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6180d606ad4c85bf8556123c6d5002" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40481 } } peers { permanent_uuid: "8655ad16feb84569ac8cdfc7a908a6b3" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 41423 } } peers { permanent_uuid: "d578ba74e64a416ab5e9b621d7e56679" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 42427 } }
I20260928 04:47:03.834544 18926 catalog_manager.cc:5495] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679 reported cstate change: term changed from 0 to 1, leader changed from <none> to d578ba74e64a416ab5e9b621d7e56679 (127.1.226.193). New cstate: current_term: 1 leader_uuid: "d578ba74e64a416ab5e9b621d7e56679" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec6180d606ad4c85bf8556123c6d5002" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8655ad16feb84569ac8cdfc7a908a6b3" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 41423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d578ba74e64a416ab5e9b621d7e56679" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 42427 } health_report { overall_health: HEALTHY } } }
I20260928 04:47:03.851146 19401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:03.848517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50016 (local address 127.1.226.193:42427)
0928 04:47:03.848577 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:03.848579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:03.848588 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:47:03.848685 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:03.848686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:03.848705 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:03.848784 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:47:03.848958 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.849464 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.850292 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:03.850493 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:03.850571 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:03.850766 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:47:03.850886 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:47:03.850976 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:03.851027 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 04:47:03.851048 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 04:47:03.853176 19205 raft_consensus.cc:1261] T 682fa61fdc3b41a8b3b16e8606a11725 P ec6180d606ad4c85bf8556123c6d5002 [term 1 FOLLOWER]: Refusing update from remote peer d578ba74e64a416ab5e9b621d7e56679: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:47:03.853219 19339 raft_consensus.cc:1261] T 682fa61fdc3b41a8b3b16e8606a11725 P 8655ad16feb84569ac8cdfc7a908a6b3 [term 1 FOLLOWER]: Refusing update from remote peer d578ba74e64a416ab5e9b621d7e56679: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:47:03.853683 19409 consensus_queue.cc:1035] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6180d606ad4c85bf8556123c6d5002" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:47:03.853835 19413 consensus_queue.cc:1035] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8655ad16feb84569ac8cdfc7a908a6b3" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 41423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:47:03.856320 19416 mvcc.cc:205] Tried to move back new op lower bound from 7334178094499856384 to 7334178094423339008. Current Snapshot: MvccSnapshot[applied={T|T < 7334178094499856384}]
I20260928 04:47:03.856892 19417 mvcc.cc:205] Tried to move back new op lower bound from 7334178094499856384 to 7334178094423339008. Current Snapshot: MvccSnapshot[applied={T|T < 7334178094499856384}]
I20260928 04:47:03.859323 18924 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:47:03.859440 18924 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:47:03.860272 18924 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 498453cced3043a9a7250b7b72382a94: Sending DeleteTablet for 3 replicas of tablet 682fa61fdc3b41a8b3b16e8606a11725
I20260928 04:47:03.860636 19185 tablet_service.cc:1514] Processing DeleteTablet for tablet 682fa61fdc3b41a8b3b16e8606a11725 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:47:03 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:40906
I20260928 04:47:03.860678 19051 tablet_service.cc:1514] Processing DeleteTablet for tablet 682fa61fdc3b41a8b3b16e8606a11725 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:47:03 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:50004
I20260928 04:47:03.860698 19319 tablet_service.cc:1514] Processing DeleteTablet for tablet 682fa61fdc3b41a8b3b16e8606a11725 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:47:03 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:33694
I20260928 04:47:03.861025 19428 tablet_replica.cc:330] T 682fa61fdc3b41a8b3b16e8606a11725 P 8655ad16feb84569ac8cdfc7a908a6b3: stopping tablet replica
I20260928 04:47:03.861023 19427 tablet_replica.cc:330] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679: stopping tablet replica
I20260928 04:47:03.861126 19427 raft_consensus.cc:2229] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:47:03.861192 19428 raft_consensus.cc:2229] T 682fa61fdc3b41a8b3b16e8606a11725 P 8655ad16feb84569ac8cdfc7a908a6b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:47:03.861263 19427 raft_consensus.cc:2258] T 682fa61fdc3b41a8b3b16e8606a11725 P d578ba74e64a416ab5e9b621d7e56679 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:47:03.861292 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 18988
I20260928 04:47:03.861756 19428 raft_consensus.cc:2258] T 682fa61fdc3b41a8b3b16e8606a11725 P 8655ad16feb84569ac8cdfc7a908a6b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:47:03.863050 19426 tablet_replica.cc:330] T 682fa61fdc3b41a8b3b16e8606a11725 P ec6180d606ad4c85bf8556123c6d5002: stopping tablet replica
I20260928 04:47:03.863046 19428 ts_tablet_manager.cc:1905] T 682fa61fdc3b41a8b3b16e8606a11725 P 8655ad16feb84569ac8cdfc7a908a6b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:47:03.863183 19426 raft_consensus.cc:2229] T 682fa61fdc3b41a8b3b16e8606a11725 P ec6180d606ad4c85bf8556123c6d5002 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:47:03.863297 19426 raft_consensus.cc:2258] T 682fa61fdc3b41a8b3b16e8606a11725 P ec6180d606ad4c85bf8556123c6d5002 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:47:03.863726 19426 ts_tablet_manager.cc:1905] T 682fa61fdc3b41a8b3b16e8606a11725 P ec6180d606ad4c85bf8556123c6d5002: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:47:03.865146 19428 ts_tablet_manager.cc:1918] T 682fa61fdc3b41a8b3b16e8606a11725 P 8655ad16feb84569ac8cdfc7a908a6b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:47:03.865231 19428 log.cc:1196] T 682fa61fdc3b41a8b3b16e8606a11725 P 8655ad16feb84569ac8cdfc7a908a6b3: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-2/wal/wals/682fa61fdc3b41a8b3b16e8606a11725
I20260928 04:47:03.865262 19426 ts_tablet_manager.cc:1918] T 682fa61fdc3b41a8b3b16e8606a11725 P ec6180d606ad4c85bf8556123c6d5002: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:47:03.865360 19426 log.cc:1196] T 682fa61fdc3b41a8b3b16e8606a11725 P ec6180d606ad4c85bf8556123c6d5002: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570585693787-1931-0/minicluster-data/ts-1/wal/wals/682fa61fdc3b41a8b3b16e8606a11725
I20260928 04:47:03.865499 19428 ts_tablet_manager.cc:1939] T 682fa61fdc3b41a8b3b16e8606a11725 P 8655ad16feb84569ac8cdfc7a908a6b3: Deleting consensus metadata
I20260928 04:47:03.865706 19426 ts_tablet_manager.cc:1939] T 682fa61fdc3b41a8b3b16e8606a11725 P ec6180d606ad4c85bf8556123c6d5002: Deleting consensus metadata
I20260928 04:47:03.865973 18912 catalog_manager.cc:4841] TS 8655ad16feb84569ac8cdfc7a908a6b3 (127.1.226.195:41423): tablet 682fa61fdc3b41a8b3b16e8606a11725 (table test-table [id=98044a5c320a40daa06fbbcb2a0726f1]) successfully deleted
I20260928 04:47:03.866117 18910 catalog_manager.cc:4841] TS ec6180d606ad4c85bf8556123c6d5002 (127.1.226.194:40481): tablet 682fa61fdc3b41a8b3b16e8606a11725 (table test-table [id=98044a5c320a40daa06fbbcb2a0726f1]) successfully deleted
W20260928 04:47:03.867525 18913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.226.193:42427 (error 108)
I20260928 04:47:03.868034 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:03.867913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:42427 (local address 127.0.0.1:50028)
0928 04:47:03.867976 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:03.868009 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.193:42427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 04:47:03.868168 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 19122
W20260928 04:47:03.868213 18913 catalog_manager.cc:4584] TS d578ba74e64a416ab5e9b621d7e56679 (127.1.226.193:42427): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 682fa61fdc3b41a8b3b16e8606a11725: Network error: Client connection negotiation failed: client connection to 127.1.226.193:42427: connect: Connection refused (error 111)
I20260928 04:47:03.873304 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 19256
I20260928 04:47:03.877547 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 18894
2026-09-28T04:47:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2108 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T04:47:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:47:03Z Disabled control of system clock
I20260928 04:47:03.904817 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:34359
--webserver_interface=127.1.226.254
--webserver_port=0
--builtin_ntp_servers=127.1.226.212:41073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:34359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 04:47:03.985041 19433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:03.985272 19433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:03.985293 19433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:03.986526 19433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:03.986572 19433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:03.986584 19433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:03.986598 19433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:03.986609 19433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:03.987947 19433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:41073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:34359
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:34359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:03.988202 19433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:03.988479 19433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:03.993422 19438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:03.993549 19441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:03.993445 19439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:03.993554 19433 server_base.cc:1034] running on GCE node
I20260928 04:47:03.993988 19433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:03.994329 19433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:03.995505 19433 hybrid_clock.cc:648] HybridClock initialized: now 1790570823995481 us; error 38 us; skew 500 ppm
I20260928 04:47:03.996778 19433 webserver.cc:460] Webserver started at http://127.1.226.254:44187/ using document root <none> and password file <none>
I20260928 04:47:03.997038 19433 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:03.997112 19433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:03.997272 19433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:03.998377 19433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "6ed47afe61514a69a60f9874060eeb03"
format_stamp: "Formatted at 2026-09-28 04:47:03 on dist-test-slave-j5c3"
server_key: "38c1bcc3a4bd4c694fa7e33facfbfb66"
server_key_iv: "af2283a832213e0e69e6b75ab4b4d74b"
server_key_version: "encryptionkey@0"
I20260928 04:47:03.998757 19433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "6ed47afe61514a69a60f9874060eeb03"
format_stamp: "Formatted at 2026-09-28 04:47:03 on dist-test-slave-j5c3"
server_key: "38c1bcc3a4bd4c694fa7e33facfbfb66"
server_key_iv: "af2283a832213e0e69e6b75ab4b4d74b"
server_key_version: "encryptionkey@0"
I20260928 04:47:04.000137 19433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:47:04.001056 19447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:04.001261 19433 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:47:04.001385 19433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "6ed47afe61514a69a60f9874060eeb03"
format_stamp: "Formatted at 2026-09-28 04:47:03 on dist-test-slave-j5c3"
server_key: "38c1bcc3a4bd4c694fa7e33facfbfb66"
server_key_iv: "af2283a832213e0e69e6b75ab4b4d74b"
server_key_version: "encryptionkey@0"
I20260928 04:47:04.001462 19433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:04.022023 19433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:04.022387 19433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:04.022560 19433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:04.026641 19433 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:34359
I20260928 04:47:04.026713 19499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:34359 every 8 connection(s)
I20260928 04:47:04.027074 19433 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:47:04.028040 19500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:04.029150 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 19433
I20260928 04:47:04.029271 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:47:04.029465 1931 external_mini_cluster.cc:1428] Setting key 12eb96e98e976643658dc91586d1d14c
I20260928 04:47:04.030772 19500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ed47afe61514a69a60f9874060eeb03: Bootstrap starting.
I20260928 04:47:04.031458 19500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ed47afe61514a69a60f9874060eeb03: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:04.031761 19500 log.cc:824] T 00000000000000000000000000000000 P 6ed47afe61514a69a60f9874060eeb03: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:04.032470 19500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ed47afe61514a69a60f9874060eeb03: No bootstrap required, opened a new log
I20260928 04:47:04.034386 19500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ed47afe61514a69a60f9874060eeb03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed47afe61514a69a60f9874060eeb03" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 34359 } }
I20260928 04:47:04.034548 19500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ed47afe61514a69a60f9874060eeb03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:04.034601 19500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ed47afe61514a69a60f9874060eeb03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed47afe61514a69a60f9874060eeb03, State: Initialized, Role: FOLLOWER
I20260928 04:47:04.034767 19500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ed47afe61514a69a60f9874060eeb03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed47afe61514a69a60f9874060eeb03" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 34359 } }
I20260928 04:47:04.034861 19500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ed47afe61514a69a60f9874060eeb03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:47:04.034921 19500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ed47afe61514a69a60f9874060eeb03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:47:04.034972 19500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ed47afe61514a69a60f9874060eeb03 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:04.035202 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:04.030020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36416 (local address 127.1.226.254:34359)
0928 04:47:04.030212 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:04.030220 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:04.030245 (+ 25us) server_negotiation.cc:407] Connection header received
0928 04:47:04.030647 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:04.030653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:04.030713 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:04.030837 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:04.031081 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.032030 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.032920 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.033346 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.033482 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:04.034325 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:04.034331 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:04.034332 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:04.034528 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:04.034564 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:04.034601 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:04.034667 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:04.034841 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 04:47:04.034948 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260928 04:47:04.035667 19500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ed47afe61514a69a60f9874060eeb03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed47afe61514a69a60f9874060eeb03" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 34359 } }
I20260928 04:47:04.035768 19500 leader_election.cc:302] T 00000000000000000000000000000000 P 6ed47afe61514a69a60f9874060eeb03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ed47afe61514a69a60f9874060eeb03; no voters:
I20260928 04:47:04.035917 19500 leader_election.cc:288] T 00000000000000000000000000000000 P 6ed47afe61514a69a60f9874060eeb03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:47:04.035980 19505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ed47afe61514a69a60f9874060eeb03 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:04.036144 19505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ed47afe61514a69a60f9874060eeb03 [term 1 LEADER]: Becoming Leader. State: Replica: 6ed47afe61514a69a60f9874060eeb03, State: Running, Role: LEADER
I20260928 04:47:04.036260 19500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ed47afe61514a69a60f9874060eeb03 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:04.036293 19505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ed47afe61514a69a60f9874060eeb03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed47afe61514a69a60f9874060eeb03" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 34359 } }
I20260928 04:47:04.036695 19507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ed47afe61514a69a60f9874060eeb03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ed47afe61514a69a60f9874060eeb03. Latest consensus state: current_term: 1 leader_uuid: "6ed47afe61514a69a60f9874060eeb03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed47afe61514a69a60f9874060eeb03" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 34359 } } }
I20260928 04:47:04.036695 19506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ed47afe61514a69a60f9874060eeb03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ed47afe61514a69a60f9874060eeb03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed47afe61514a69a60f9874060eeb03" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 34359 } } }
I20260928 04:47:04.036808 19507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ed47afe61514a69a60f9874060eeb03 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:04.036815 19506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ed47afe61514a69a60f9874060eeb03 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:04.037299 19513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:47:04.037837 19513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:47:04.039362 19513 catalog_manager.cc:1347] Generated new cluster ID: 8072fe5efa6e4683b0358678affca4d3
I20260928 04:47:04.039422 19513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:47:04.083952 19513 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:47:04.084746 19513 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:47:04.092428 19513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ed47afe61514a69a60f9874060eeb03: Generated new TSK 0
I20260928 04:47:04.092635 19513 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:47:04.103797 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:0
--local_ip_for_outbound_sockets=127.1.226.193
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:34359
--builtin_ntp_servers=127.1.226.212:41073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 04:47:04.185961 19524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:04.186213 19524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:04.186250 19524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:04.187492 19524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:04.187557 19524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:04.187603 19524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:47:04.189065 19524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:41073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:34359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:04.189335 19524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:04.189654 19524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:04.194721 19529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:04.194796 19524 server_base.cc:1034] running on GCE node
W20260928 04:47:04.194721 19532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:04.194751 19530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:04.195192 19524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:04.195482 19524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:04.196630 19524 hybrid_clock.cc:648] HybridClock initialized: now 1790570824196612 us; error 42 us; skew 500 ppm
I20260928 04:47:04.198084 19524 webserver.cc:460] Webserver started at http://127.1.226.193:46573/ using document root <none> and password file <none>
I20260928 04:47:04.198398 19524 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:04.198470 19524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:04.198642 19524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:04.199744 19524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-0/data/instance:
uuid: "028652419b1e44138f41e36ab2143c24"
format_stamp: "Formatted at 2026-09-28 04:47:04 on dist-test-slave-j5c3"
server_key: "1099bcbc65b1069f2de685d0950cfaf3"
server_key_iv: "97bf7d7637fa6de0bac14c143ed8e0ee"
server_key_version: "encryptionkey@0"
I20260928 04:47:04.200127 19524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance:
uuid: "028652419b1e44138f41e36ab2143c24"
format_stamp: "Formatted at 2026-09-28 04:47:04 on dist-test-slave-j5c3"
server_key: "1099bcbc65b1069f2de685d0950cfaf3"
server_key_iv: "97bf7d7637fa6de0bac14c143ed8e0ee"
server_key_version: "encryptionkey@0"
I20260928 04:47:04.201510 19524 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:04.202368 19538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:04.202569 19524 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:04.202642 19524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "028652419b1e44138f41e36ab2143c24"
format_stamp: "Formatted at 2026-09-28 04:47:04 on dist-test-slave-j5c3"
server_key: "1099bcbc65b1069f2de685d0950cfaf3"
server_key_iv: "97bf7d7637fa6de0bac14c143ed8e0ee"
server_key_version: "encryptionkey@0"
I20260928 04:47:04.202729 19524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:04.211825 19524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:04.212132 19524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:04.212347 19524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:04.212579 19524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:47:04.212935 19524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:47:04.212990 19524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:04.213033 19524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:47:04.213060 19524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:04.219107 19524 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:37941
I20260928 04:47:04.219147 19651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:37941 every 8 connection(s)
I20260928 04:47:04.219493 19524 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
I20260928 04:47:04.224879 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:04.220281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:58193 (local address 127.1.226.254:34359)
0928 04:47:04.220342 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:04.220345 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:04.220605 (+ 260us) server_negotiation.cc:407] Connection header received
0928 04:47:04.221148 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:04.221149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:04.221163 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:04.221209 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:04.221892 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.222406 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.223421 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.223640 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.223721 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:04.224584 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:04.224586 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:04.224586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:04.224591 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:04.224606 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:04.224610 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:04.224666 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:04.224759 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 04:47:04.224790 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 04:47:04.225102 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:04.220328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:34359 (local address 127.1.226.193:58193)
0928 04:47:04.220481 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:04.220515 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:04.221028 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:04.221299 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:04.221302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:04.221310 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:04.221795 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:04.221800 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.222508 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.222509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:04.223325 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:04.223327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.223729 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.223730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:04.223865 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:04.224440 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:47:04.224441 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:04.224469 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:47:04.224493 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:04.224669 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:04.224674 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:04.224677 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:47:04.224761 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 04:47:04.224865 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":65,"threads_started":1}
I20260928 04:47:04.225569 19652 heartbeater.cc:347] Connected to a master server at 127.1.226.254:34359
I20260928 04:47:04.225651 19652 heartbeater.cc:464] Registering TS with master...
I20260928 04:47:04.225857 19652 heartbeater.cc:511] Master 127.1.226.254:34359 requested a full tablet report, sending...
I20260928 04:47:04.226353 19464 ts_manager.cc:194] Registered new tserver with Master: 028652419b1e44138f41e36ab2143c24 (127.1.226.193:37941)
I20260928 04:47:04.227089 19464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.226.193:58193
I20260928 04:47:04.228951 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 19524
I20260928 04:47:04.229035 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance
I20260928 04:47:04.229192 1931 external_mini_cluster.cc:1428] Setting key 3ab396964f9b2cb507ccaffabf26d0d9
I20260928 04:47:04.230069 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:0
--local_ip_for_outbound_sockets=127.1.226.194
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:34359
--builtin_ntp_servers=127.1.226.212:41073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 04:47:04.312536 19655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:04.312816 19655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:04.312844 19655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:04.313998 19655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:04.314035 19655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:04.314060 19655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:47:04.315367 19655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:41073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:34359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:04.315568 19655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:04.315834 19655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:04.321017 19663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:04.321017 19660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:04.321043 19661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:04.321305 19655 server_base.cc:1034] running on GCE node
I20260928 04:47:04.321499 19655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:04.321755 19655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:04.322898 19655 hybrid_clock.cc:648] HybridClock initialized: now 1790570824322880 us; error 28 us; skew 500 ppm
I20260928 04:47:04.324344 19655 webserver.cc:460] Webserver started at http://127.1.226.194:39895/ using document root <none> and password file <none>
I20260928 04:47:04.324594 19655 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:04.324666 19655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:04.324821 19655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:04.325901 19655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-1/data/instance:
uuid: "2aed97034fd148e09d11b9247dd5823d"
format_stamp: "Formatted at 2026-09-28 04:47:04 on dist-test-slave-j5c3"
server_key: "8db013540f6a558009db04dcf94c2efb"
server_key_iv: "ceeb4a33d078410c19ec4ab0e074d0e4"
server_key_version: "encryptionkey@0"
I20260928 04:47:04.326318 19655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance:
uuid: "2aed97034fd148e09d11b9247dd5823d"
format_stamp: "Formatted at 2026-09-28 04:47:04 on dist-test-slave-j5c3"
server_key: "8db013540f6a558009db04dcf94c2efb"
server_key_iv: "ceeb4a33d078410c19ec4ab0e074d0e4"
server_key_version: "encryptionkey@0"
I20260928 04:47:04.327749 19655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:47:04.328588 19669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:04.328783 19655 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:04.328897 19655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "2aed97034fd148e09d11b9247dd5823d"
format_stamp: "Formatted at 2026-09-28 04:47:04 on dist-test-slave-j5c3"
server_key: "8db013540f6a558009db04dcf94c2efb"
server_key_iv: "ceeb4a33d078410c19ec4ab0e074d0e4"
server_key_version: "encryptionkey@0"
I20260928 04:47:04.328987 19655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:04.339205 19655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:04.339540 19655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:04.339697 19655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:04.339931 19655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:47:04.340341 19655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:47:04.340396 19655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:04.340438 19655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:47:04.340464 19655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:04.346683 19655 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:36853
I20260928 04:47:04.346732 19782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:36853 every 8 connection(s)
I20260928 04:47:04.347087 19655 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
I20260928 04:47:04.352559 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:04.347848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:43821 (local address 127.1.226.254:34359)
0928 04:47:04.347915 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:04.347917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:04.348140 (+ 223us) server_negotiation.cc:407] Connection header received
0928 04:47:04.348697 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:04.348699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:04.348711 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:04.348763 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:04.349518 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.350042 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.351055 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.351268 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.351345 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:04.352247 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:04.352249 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:04.352249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:04.352254 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:04.352275 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:04.352280 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:04.352344 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:04.352449 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 04:47:04.352477 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 04:47:04.352756 19785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:04.347842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:34359 (local address 127.1.226.194:43821)
0928 04:47:04.348014 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:04.348049 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:04.348589 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:04.348852 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:04.348855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:04.348863 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:04.349363 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:04.349368 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.350143 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.350144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:04.350960 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:04.350962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.351365 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.351366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:04.351497 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:04.352075 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:47:04.352075 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:04.352104 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:47:04.352126 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:04.352345 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:04.352351 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:04.352354 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:47:04.352437 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 04:47:04.352540 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
I20260928 04:47:04.353240 19783 heartbeater.cc:347] Connected to a master server at 127.1.226.254:34359
I20260928 04:47:04.353324 19783 heartbeater.cc:464] Registering TS with master...
I20260928 04:47:04.353525 19783 heartbeater.cc:511] Master 127.1.226.254:34359 requested a full tablet report, sending...
I20260928 04:47:04.353929 19464 ts_manager.cc:194] Registered new tserver with Master: 2aed97034fd148e09d11b9247dd5823d (127.1.226.194:36853)
I20260928 04:47:04.354305 19464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.226.194:43821
I20260928 04:47:04.354919 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 19655
I20260928 04:47:04.355010 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance
I20260928 04:47:04.355247 1931 external_mini_cluster.cc:1428] Setting key a79a397e25407faa23f12ef6d36604d1
I20260928 04:47:04.356220 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:0
--local_ip_for_outbound_sockets=127.1.226.195
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:34359
--builtin_ntp_servers=127.1.226.212:41073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 04:47:04.440299 19786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:04.440580 19786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:04.440604 19786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:04.441802 19786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:04.441846 19786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:04.441872 19786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:47:04.443295 19786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:41073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:34359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:04.443571 19786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:04.443902 19786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:04.449131 19792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:04.449082 19794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:04.449137 19791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:04.449285 19786 server_base.cc:1034] running on GCE node
I20260928 04:47:04.449601 19786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:04.449829 19786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:04.450994 19786 hybrid_clock.cc:648] HybridClock initialized: now 1790570824450981 us; error 31 us; skew 500 ppm
I20260928 04:47:04.452416 19786 webserver.cc:460] Webserver started at http://127.1.226.195:35475/ using document root <none> and password file <none>
I20260928 04:47:04.452678 19786 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:04.452737 19786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:04.452907 19786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:04.454012 19786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-2/data/instance:
uuid: "b5311372eb7445668d1ab4e4d515dc9f"
format_stamp: "Formatted at 2026-09-28 04:47:04 on dist-test-slave-j5c3"
server_key: "856ac58b28f48469134a4ce3a18932c6"
server_key_iv: "5f92b70bc1bec6862e2c848bddcc3425"
server_key_version: "encryptionkey@0"
I20260928 04:47:04.454389 19786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance:
uuid: "b5311372eb7445668d1ab4e4d515dc9f"
format_stamp: "Formatted at 2026-09-28 04:47:04 on dist-test-slave-j5c3"
server_key: "856ac58b28f48469134a4ce3a18932c6"
server_key_iv: "5f92b70bc1bec6862e2c848bddcc3425"
server_key_version: "encryptionkey@0"
I20260928 04:47:04.455794 19786 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:47:04.456660 19800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:04.456835 19786 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:47:04.456933 19786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "b5311372eb7445668d1ab4e4d515dc9f"
format_stamp: "Formatted at 2026-09-28 04:47:04 on dist-test-slave-j5c3"
server_key: "856ac58b28f48469134a4ce3a18932c6"
server_key_iv: "5f92b70bc1bec6862e2c848bddcc3425"
server_key_version: "encryptionkey@0"
I20260928 04:47:04.456993 19786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:04.463900 19786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:04.464145 19786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:04.464315 19786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:04.464488 19786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:47:04.464807 19786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:47:04.464845 19786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:04.464876 19786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:47:04.464895 19786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:04.470692 19786 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:41689
I20260928 04:47:04.470726 19913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:41689 every 8 connection(s)
I20260928 04:47:04.471069 19786 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
I20260928 04:47:04.476372 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:04.471840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:44219 (local address 127.1.226.254:34359)
0928 04:47:04.471919 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:04.471921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:04.472238 (+ 317us) server_negotiation.cc:407] Connection header received
0928 04:47:04.472762 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:04.472764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:04.472776 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:04.472820 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:04.473490 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.473976 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.474930 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.475117 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.475196 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:04.476038 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:04.476039 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:04.476039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:04.476043 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:04.476058 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:04.476062 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:04.476117 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:04.476276 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 04:47:04.476304 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 04:47:04.476549 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:04.471794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:34359 (local address 127.1.226.195:44219)
0928 04:47:04.472044 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:04.472080 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:04.472653 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:04.472890 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:04.472893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:04.472901 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:04.473413 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:04.473417 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.474065 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.474066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:04.474843 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:04.474845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.475201 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.475202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:04.475321 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:04.475888 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:47:04.475888 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:04.475916 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:47:04.475938 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:04.476128 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:04.476132 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:04.476135 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:47:04.476238 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 04:47:04.476328 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":108,"threads_started":1}
I20260928 04:47:04.477100 19914 heartbeater.cc:347] Connected to a master server at 127.1.226.254:34359
I20260928 04:47:04.477185 19914 heartbeater.cc:464] Registering TS with master...
I20260928 04:47:04.477392 19914 heartbeater.cc:511] Master 127.1.226.254:34359 requested a full tablet report, sending...
I20260928 04:47:04.477866 19464 ts_manager.cc:194] Registered new tserver with Master: b5311372eb7445668d1ab4e4d515dc9f (127.1.226.195:41689)
I20260928 04:47:04.478226 19464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.226.195:44219
I20260928 04:47:04.481133 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 19786
I20260928 04:47:04.481231 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance
I20260928 04:47:04.481408 1931 external_mini_cluster.cc:1428] Setting key af40efa102deae43396066c98ba318ec
I20260928 04:47:04.482461 1931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:47:04.486876 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:04.483999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36422 (local address 127.1.226.254:34359)
0928 04:47:04.484059 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:04.484061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:04.484188 (+ 127us) server_negotiation.cc:407] Connection header received
0928 04:47:04.484311 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:04.484312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:04.484321 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:04.484380 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:04.484513 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.485011 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.485812 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.485993 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.486047 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:04.486693 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:04.486695 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:04.486695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:04.486699 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:04.486712 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:04.486716 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:04.486759 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:04.486793 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 04:47:04.486814 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 04:47:04.488436 19464 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:47:04.490063 19464 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:47:04.496903 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:04.492674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44418 (local address 127.1.226.193:37941)
0928 04:47:04.493141 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:04.493145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:04.493155 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:47:04.493216 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:04.493219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:04.493234 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:04.493316 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:04.493491 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.494564 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.495711 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.496469 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.496547 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:04.496641 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:04.496768 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 04:47:04.496815 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":70,"threads_started":1}
I20260928 04:47:04.496897 19922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:04.492549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:37941 (local address 127.0.0.1:44418)
0928 04:47:04.492849 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:04.492894 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:04.492964 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:04.493384 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:04.493386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:04.493388 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:04.493421 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:04.493422 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.494649 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.494650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:04.495621 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:04.495624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.496580 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.496581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:04.496653 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:04.496666 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 04:47:04.496741 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 04:47:04.496797 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":102,"threads_started":1}
I20260928 04:47:04.496908 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:04.492707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:41689 (local address 127.0.0.1:42524)
0928 04:47:04.493321 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:04.493331 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:04.493390 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:04.493642 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:04.493642 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:04.493644 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:04.493676 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:04.493677 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.494484 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.494485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:04.495813 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:04.495816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.496709 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.496710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:04.496765 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:04.496768 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:47:04.496814 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 04:47:04.496845 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":60,"thread_start_us":68,"threads_started":1}
I20260928 04:47:04.497088 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:04.492885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42524 (local address 127.1.226.195:41689)
0928 04:47:04.493036 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:04.493039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:04.493504 (+ 465us) server_negotiation.cc:407] Connection header received
0928 04:47:04.493545 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:04.493546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:04.493559 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:04.493645 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:04.493730 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.494398 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.495919 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.496486 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.496589 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:04.496677 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:04.496944 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 04:47:04.496989 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":94,"threads_started":1}
I20260928 04:47:04.497442 19924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:04.492598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37008 (local address 127.1.226.194:36853)
0928 04:47:04.492900 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:04.492906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:04.492951 (+ 45us) server_negotiation.cc:407] Connection header received
0928 04:47:04.493080 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:04.493083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:04.493098 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:04.493221 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:04.493397 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.494538 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.495905 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.496612 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.496673 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:04.496774 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:04.497297 (+ 523us) server_negotiation.cc:299] Negotiation successful
0928 04:47:04.497347 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":149,"threads_started":1}
I20260928 04:47:04.497661 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:04.492471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:36853 (local address 127.0.0.1:37008)
0928 04:47:04.492849 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:04.492894 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:04.492964 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:04.493237 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:04.493240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:04.493242 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:04.493329 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:04.493332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.494629 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.494630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:04.495817 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:04.495819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.496808 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.496809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:04.496872 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:04.496876 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:47:04.496930 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 04:47:04.496974 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":40,"thread_start_us":86,"threads_started":1}
I20260928 04:47:04.497774 19586 tablet_service.cc:1467] Processing CreateTablet for tablet 57cd3800b8614aea8b5a5c111bf899ce (DEFAULT_TABLE table=test-table [id=40c615d7798442afb1b96948fdb38741]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:47:04.497917 19848 tablet_service.cc:1467] Processing CreateTablet for tablet 57cd3800b8614aea8b5a5c111bf899ce (DEFAULT_TABLE table=test-table [id=40c615d7798442afb1b96948fdb38741]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:47:04.498165 19586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57cd3800b8614aea8b5a5c111bf899ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:04.498190 19848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57cd3800b8614aea8b5a5c111bf899ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:04.498427 19717 tablet_service.cc:1467] Processing CreateTablet for tablet 57cd3800b8614aea8b5a5c111bf899ce (DEFAULT_TABLE table=test-table [id=40c615d7798442afb1b96948fdb38741]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:47:04.498663 19717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57cd3800b8614aea8b5a5c111bf899ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:04.501525 19928 tablet_bootstrap.cc:492] T 57cd3800b8614aea8b5a5c111bf899ce P 2aed97034fd148e09d11b9247dd5823d: Bootstrap starting.
I20260928 04:47:04.502055 19928 tablet_bootstrap.cc:654] T 57cd3800b8614aea8b5a5c111bf899ce P 2aed97034fd148e09d11b9247dd5823d: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:04.502285 19928 log.cc:824] T 57cd3800b8614aea8b5a5c111bf899ce P 2aed97034fd148e09d11b9247dd5823d: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:04.502530 19929 tablet_bootstrap.cc:492] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f: Bootstrap starting.
I20260928 04:47:04.503070 19928 tablet_bootstrap.cc:492] T 57cd3800b8614aea8b5a5c111bf899ce P 2aed97034fd148e09d11b9247dd5823d: No bootstrap required, opened a new log
I20260928 04:47:04.503118 19929 tablet_bootstrap.cc:654] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:04.503147 19928 ts_tablet_manager.cc:1397] T 57cd3800b8614aea8b5a5c111bf899ce P 2aed97034fd148e09d11b9247dd5823d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:47:04.503404 19929 log.cc:824] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:04.503759 19930 tablet_bootstrap.cc:492] T 57cd3800b8614aea8b5a5c111bf899ce P 028652419b1e44138f41e36ab2143c24: Bootstrap starting.
I20260928 04:47:04.504593 19930 tablet_bootstrap.cc:654] T 57cd3800b8614aea8b5a5c111bf899ce P 028652419b1e44138f41e36ab2143c24: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:04.504915 19930 log.cc:824] T 57cd3800b8614aea8b5a5c111bf899ce P 028652419b1e44138f41e36ab2143c24: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:04.505451 19929 tablet_bootstrap.cc:492] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f: No bootstrap required, opened a new log
I20260928 04:47:04.505520 19929 ts_tablet_manager.cc:1397] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 04:47:04.506026 19928 raft_consensus.cc:348] T 57cd3800b8614aea8b5a5c111bf899ce P 2aed97034fd148e09d11b9247dd5823d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aed97034fd148e09d11b9247dd5823d" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36853 } } peers { permanent_uuid: "b5311372eb7445668d1ab4e4d515dc9f" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 41689 } } peers { permanent_uuid: "028652419b1e44138f41e36ab2143c24" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37941 } }
I20260928 04:47:04.506273 19928 raft_consensus.cc:374] T 57cd3800b8614aea8b5a5c111bf899ce P 2aed97034fd148e09d11b9247dd5823d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:04.506332 19928 raft_consensus.cc:729] T 57cd3800b8614aea8b5a5c111bf899ce P 2aed97034fd148e09d11b9247dd5823d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aed97034fd148e09d11b9247dd5823d, State: Initialized, Role: FOLLOWER
I20260928 04:47:04.506469 19928 consensus_queue.cc:260] T 57cd3800b8614aea8b5a5c111bf899ce P 2aed97034fd148e09d11b9247dd5823d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aed97034fd148e09d11b9247dd5823d" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36853 } } peers { permanent_uuid: "b5311372eb7445668d1ab4e4d515dc9f" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 41689 } } peers { permanent_uuid: "028652419b1e44138f41e36ab2143c24" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37941 } }
I20260928 04:47:04.506727 19928 ts_tablet_manager.cc:1428] T 57cd3800b8614aea8b5a5c111bf899ce P 2aed97034fd148e09d11b9247dd5823d: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260928 04:47:04.506897 19783 heartbeater.cc:503] Master 127.1.226.254:34359 was elected leader, sending a full tablet report...
I20260928 04:47:04.507799 19929 raft_consensus.cc:348] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aed97034fd148e09d11b9247dd5823d" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36853 } } peers { permanent_uuid: "b5311372eb7445668d1ab4e4d515dc9f" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 41689 } } peers { permanent_uuid: "028652419b1e44138f41e36ab2143c24" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37941 } }
I20260928 04:47:04.507944 19929 raft_consensus.cc:374] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:04.507990 19929 raft_consensus.cc:729] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5311372eb7445668d1ab4e4d515dc9f, State: Initialized, Role: FOLLOWER
I20260928 04:47:04.508105 19929 consensus_queue.cc:260] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aed97034fd148e09d11b9247dd5823d" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36853 } } peers { permanent_uuid: "b5311372eb7445668d1ab4e4d515dc9f" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 41689 } } peers { permanent_uuid: "028652419b1e44138f41e36ab2143c24" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37941 } }
I20260928 04:47:04.508482 19929 ts_tablet_manager.cc:1428] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:47:04.508517 19914 heartbeater.cc:503] Master 127.1.226.254:34359 was elected leader, sending a full tablet report...
I20260928 04:47:04.508879 19930 tablet_bootstrap.cc:492] T 57cd3800b8614aea8b5a5c111bf899ce P 028652419b1e44138f41e36ab2143c24: No bootstrap required, opened a new log
I20260928 04:47:04.508956 19930 ts_tablet_manager.cc:1397] T 57cd3800b8614aea8b5a5c111bf899ce P 028652419b1e44138f41e36ab2143c24: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 04:47:04.510895 19930 raft_consensus.cc:348] T 57cd3800b8614aea8b5a5c111bf899ce P 028652419b1e44138f41e36ab2143c24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aed97034fd148e09d11b9247dd5823d" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36853 } } peers { permanent_uuid: "b5311372eb7445668d1ab4e4d515dc9f" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 41689 } } peers { permanent_uuid: "028652419b1e44138f41e36ab2143c24" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37941 } }
I20260928 04:47:04.511018 19930 raft_consensus.cc:374] T 57cd3800b8614aea8b5a5c111bf899ce P 028652419b1e44138f41e36ab2143c24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:04.511039 19930 raft_consensus.cc:729] T 57cd3800b8614aea8b5a5c111bf899ce P 028652419b1e44138f41e36ab2143c24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 028652419b1e44138f41e36ab2143c24, State: Initialized, Role: FOLLOWER
I20260928 04:47:04.511147 19930 consensus_queue.cc:260] T 57cd3800b8614aea8b5a5c111bf899ce P 028652419b1e44138f41e36ab2143c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aed97034fd148e09d11b9247dd5823d" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36853 } } peers { permanent_uuid: "b5311372eb7445668d1ab4e4d515dc9f" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 41689 } } peers { permanent_uuid: "028652419b1e44138f41e36ab2143c24" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37941 } }
I20260928 04:47:04.511400 19930 ts_tablet_manager.cc:1428] T 57cd3800b8614aea8b5a5c111bf899ce P 028652419b1e44138f41e36ab2143c24: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 04:47:04.511471 19652 heartbeater.cc:503] Master 127.1.226.254:34359 was elected leader, sending a full tablet report...
I20260928 04:47:04.526046 19935 raft_consensus.cc:482] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:47:04.526229 19935 raft_consensus.cc:504] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aed97034fd148e09d11b9247dd5823d" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36853 } } peers { permanent_uuid: "b5311372eb7445668d1ab4e4d515dc9f" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 41689 } } peers { permanent_uuid: "028652419b1e44138f41e36ab2143c24" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37941 } }
I20260928 04:47:04.526599 19935 leader_election.cc:288] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2aed97034fd148e09d11b9247dd5823d (127.1.226.194:36853), 028652419b1e44138f41e36ab2143c24 (127.1.226.193:37941)
I20260928 04:47:04.530100 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:04.526899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:32817 (local address 127.1.226.193:37941)
0928 04:47:04.526953 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:04.526955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:04.526968 (+ 13us) server_negotiation.cc:407] Connection header received
0928 04:47:04.527002 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:04.527003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:04.527016 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:04.527088 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:04.527309 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.528091 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.529310 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.529716 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.529817 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:04.529828 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:04.529973 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 04:47:04.530024 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 04:47:04.530122 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:04.526813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:37941 (local address 127.1.226.195:32817)
0928 04:47:04.526867 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:04.526880 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:04.526929 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:04.527115 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:04.527117 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:04.527119 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:04.527223 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:04.527225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.528237 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.528238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:04.529217 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:04.529219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.529817 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.529818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:04.529887 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:04.529891 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:47:04.529966 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 04:47:04.530009 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 04:47:04.530141 19937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:04.526880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:36853 (local address 127.1.226.195:48567)
0928 04:47:04.527104 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:04.527114 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:04.527170 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:04.527308 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:04.527309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:04.527310 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:04.527372 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:04.527373 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.528269 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.528270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:04.529235 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:04.529238 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.529821 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.529822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:04.529914 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:04.529919 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:47:04.529988 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 04:47:04.530029 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":136,"threads_started":1}
I20260928 04:47:04.530123 19924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:04.526969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:48567 (local address 127.1.226.194:36853)
0928 04:47:04.527015 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:04.527017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:04.527193 (+ 176us) server_negotiation.cc:407] Connection header received
0928 04:47:04.527223 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:04.527224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:04.527232 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:04.527305 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:04.527437 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.528138 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.529339 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.529719 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.529821 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:04.529830 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:04.530008 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 04:47:04.530052 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:47:04.530478 19606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57cd3800b8614aea8b5a5c111bf899ce" candidate_uuid: "b5311372eb7445668d1ab4e4d515dc9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "028652419b1e44138f41e36ab2143c24" is_pre_election: true
I20260928 04:47:04.530623 19606 raft_consensus.cc:2454] T 57cd3800b8614aea8b5a5c111bf899ce P 028652419b1e44138f41e36ab2143c24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5311372eb7445668d1ab4e4d515dc9f in term 0.
I20260928 04:47:04.530805 19737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57cd3800b8614aea8b5a5c111bf899ce" candidate_uuid: "b5311372eb7445668d1ab4e4d515dc9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aed97034fd148e09d11b9247dd5823d" is_pre_election: true
I20260928 04:47:04.530897 19802 leader_election.cc:302] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 028652419b1e44138f41e36ab2143c24, b5311372eb7445668d1ab4e4d515dc9f; no voters:
I20260928 04:47:04.530957 19737 raft_consensus.cc:2454] T 57cd3800b8614aea8b5a5c111bf899ce P 2aed97034fd148e09d11b9247dd5823d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5311372eb7445668d1ab4e4d515dc9f in term 0.
I20260928 04:47:04.531023 19935 raft_consensus.cc:2781] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:47:04.531088 19935 raft_consensus.cc:482] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:47:04.531124 19935 raft_consensus.cc:3037] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:04.531823 19935 raft_consensus.cc:504] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aed97034fd148e09d11b9247dd5823d" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36853 } } peers { permanent_uuid: "b5311372eb7445668d1ab4e4d515dc9f" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 41689 } } peers { permanent_uuid: "028652419b1e44138f41e36ab2143c24" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37941 } }
I20260928 04:47:04.531996 19935 leader_election.cc:288] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f [CANDIDATE]: Term 1 election: Requested vote from peers 2aed97034fd148e09d11b9247dd5823d (127.1.226.194:36853), 028652419b1e44138f41e36ab2143c24 (127.1.226.193:37941)
I20260928 04:47:04.532217 19737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57cd3800b8614aea8b5a5c111bf899ce" candidate_uuid: "b5311372eb7445668d1ab4e4d515dc9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aed97034fd148e09d11b9247dd5823d"
I20260928 04:47:04.532315 19737 raft_consensus.cc:3037] T 57cd3800b8614aea8b5a5c111bf899ce P 2aed97034fd148e09d11b9247dd5823d [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:04.533180 19737 raft_consensus.cc:2454] T 57cd3800b8614aea8b5a5c111bf899ce P 2aed97034fd148e09d11b9247dd5823d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5311372eb7445668d1ab4e4d515dc9f in term 1.
I20260928 04:47:04.533352 19606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57cd3800b8614aea8b5a5c111bf899ce" candidate_uuid: "b5311372eb7445668d1ab4e4d515dc9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "028652419b1e44138f41e36ab2143c24"
I20260928 04:47:04.533473 19801 leader_election.cc:302] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aed97034fd148e09d11b9247dd5823d, b5311372eb7445668d1ab4e4d515dc9f; no voters:
I20260928 04:47:04.533478 19606 raft_consensus.cc:3037] T 57cd3800b8614aea8b5a5c111bf899ce P 028652419b1e44138f41e36ab2143c24 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:04.533565 19935 raft_consensus.cc:2781] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:04.533830 19935 raft_consensus.cc:686] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f [term 1 LEADER]: Becoming Leader. State: Replica: b5311372eb7445668d1ab4e4d515dc9f, State: Running, Role: LEADER
I20260928 04:47:04.533936 19935 consensus_queue.cc:237] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aed97034fd148e09d11b9247dd5823d" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36853 } } peers { permanent_uuid: "b5311372eb7445668d1ab4e4d515dc9f" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 41689 } } peers { permanent_uuid: "028652419b1e44138f41e36ab2143c24" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37941 } }
I20260928 04:47:04.534384 19606 raft_consensus.cc:2454] T 57cd3800b8614aea8b5a5c111bf899ce P 028652419b1e44138f41e36ab2143c24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5311372eb7445668d1ab4e4d515dc9f in term 1.
I20260928 04:47:04.534835 19464 catalog_manager.cc:5495] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f reported cstate change: term changed from 0 to 1, leader changed from <none> to b5311372eb7445668d1ab4e4d515dc9f (127.1.226.195). New cstate: current_term: 1 leader_uuid: "b5311372eb7445668d1ab4e4d515dc9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aed97034fd148e09d11b9247dd5823d" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5311372eb7445668d1ab4e4d515dc9f" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 41689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "028652419b1e44138f41e36ab2143c24" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37941 } health_report { overall_health: UNKNOWN } } }
I20260928 04:47:04.543511 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:04.540911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42530 (local address 127.1.226.195:41689)
0928 04:47:04.540970 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:04.540972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:04.541026 (+ 54us) server_negotiation.cc:407] Connection header received
0928 04:47:04.541098 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:04.541098 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:04.541117 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:04.541182 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:47:04.541324 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.541803 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.542623 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:04.542873 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:04.542945 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:04.543126 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:47:04.543237 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:47:04.543308 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:04.543344 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 04:47:04.543370 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 04:47:04.545665 19737 raft_consensus.cc:1261] T 57cd3800b8614aea8b5a5c111bf899ce P 2aed97034fd148e09d11b9247dd5823d [term 1 FOLLOWER]: Refusing update from remote peer b5311372eb7445668d1ab4e4d515dc9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:47:04.545687 19606 raft_consensus.cc:1261] T 57cd3800b8614aea8b5a5c111bf899ce P 028652419b1e44138f41e36ab2143c24 [term 1 FOLLOWER]: Refusing update from remote peer b5311372eb7445668d1ab4e4d515dc9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:47:04.546073 19938 consensus_queue.cc:1035] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "028652419b1e44138f41e36ab2143c24" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:47:04.546213 19935 consensus_queue.cc:1035] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aed97034fd148e09d11b9247dd5823d" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 36853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:47:04.548825 19941 mvcc.cc:205] Tried to move back new op lower bound from 7334178097336385536 to 7334178097291923456. Current Snapshot: MvccSnapshot[applied={T|T < 7334178097336385536}]
I20260928 04:47:04.549090 19942 mvcc.cc:205] Tried to move back new op lower bound from 7334178097336385536 to 7334178097291923456. Current Snapshot: MvccSnapshot[applied={T|T < 7334178097336385536}]
I20260928 04:47:04.549288 19943 mvcc.cc:205] Tried to move back new op lower bound from 7334178097336385536 to 7334178097291923456. Current Snapshot: MvccSnapshot[applied={T|T < 7334178097336385536}]
I20260928 04:47:04.552003 19463 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:47:04.552105 19463 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:47:04.552976 19463 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ed47afe61514a69a60f9874060eeb03: Sending DeleteTablet for 3 replicas of tablet 57cd3800b8614aea8b5a5c111bf899ce
I20260928 04:47:04.553424 19848 tablet_service.cc:1514] Processing DeleteTablet for tablet 57cd3800b8614aea8b5a5c111bf899ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:47:04 UTC) from {username='slave'} at 127.0.0.1:42524
I20260928 04:47:04.553436 19717 tablet_service.cc:1514] Processing DeleteTablet for tablet 57cd3800b8614aea8b5a5c111bf899ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:47:04 UTC) from {username='slave'} at 127.0.0.1:37008
I20260928 04:47:04.553458 19586 tablet_service.cc:1514] Processing DeleteTablet for tablet 57cd3800b8614aea8b5a5c111bf899ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:47:04 UTC) from {username='slave'} at 127.0.0.1:44418
I20260928 04:47:04.553674 19951 tablet_replica.cc:330] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f: stopping tablet replica
I20260928 04:47:04.553857 19951 raft_consensus.cc:2229] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:47:04.553910 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 19524
I20260928 04:47:04.554049 19951 raft_consensus.cc:2258] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:47:04.554221 19952 tablet_replica.cc:330] T 57cd3800b8614aea8b5a5c111bf899ce P 2aed97034fd148e09d11b9247dd5823d: stopping tablet replica
I20260928 04:47:04.554510 19952 raft_consensus.cc:2229] T 57cd3800b8614aea8b5a5c111bf899ce P 2aed97034fd148e09d11b9247dd5823d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:47:04.554626 19952 raft_consensus.cc:2258] T 57cd3800b8614aea8b5a5c111bf899ce P 2aed97034fd148e09d11b9247dd5823d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:47:04.555670 19951 ts_tablet_manager.cc:1905] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:47:04.555811 19952 ts_tablet_manager.cc:1905] T 57cd3800b8614aea8b5a5c111bf899ce P 2aed97034fd148e09d11b9247dd5823d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:47:04.557710 19952 ts_tablet_manager.cc:1918] T 57cd3800b8614aea8b5a5c111bf899ce P 2aed97034fd148e09d11b9247dd5823d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:47:04.557873 19952 log.cc:1196] T 57cd3800b8614aea8b5a5c111bf899ce P 2aed97034fd148e09d11b9247dd5823d: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-1/wal/wals/57cd3800b8614aea8b5a5c111bf899ce
I20260928 04:47:04.558212 19951 ts_tablet_manager.cc:1918] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:47:04.558320 19951 log.cc:1196] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570585693787-1931-0/minicluster-data/ts-2/wal/wals/57cd3800b8614aea8b5a5c111bf899ce
I20260928 04:47:04.558317 19952 ts_tablet_manager.cc:1939] T 57cd3800b8614aea8b5a5c111bf899ce P 2aed97034fd148e09d11b9247dd5823d: Deleting consensus metadata
I20260928 04:47:04.558627 19951 ts_tablet_manager.cc:1939] T 57cd3800b8614aea8b5a5c111bf899ce P b5311372eb7445668d1ab4e4d515dc9f: Deleting consensus metadata
I20260928 04:47:04.565936 19448 catalog_manager.cc:4841] TS b5311372eb7445668d1ab4e4d515dc9f (127.1.226.195:41689): tablet 57cd3800b8614aea8b5a5c111bf899ce (table test-table [id=40c615d7798442afb1b96948fdb38741]) successfully deleted
I20260928 04:47:04.566076 19448 catalog_manager.cc:4841] TS 2aed97034fd148e09d11b9247dd5823d (127.1.226.194:36853): tablet 57cd3800b8614aea8b5a5c111bf899ce (table test-table [id=40c615d7798442afb1b96948fdb38741]) successfully deleted
W20260928 04:47:04.567847 19449 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.226.193:37941 (error 108)
I20260928 04:47:04.568575 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:04.568423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:37941 (local address 127.0.0.1:44422)
0928 04:47:04.568489 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:04.568538 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.193:37941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 04:47:04.568713 19449 catalog_manager.cc:4584] TS 028652419b1e44138f41e36ab2143c24 (127.1.226.193:37941): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 57cd3800b8614aea8b5a5c111bf899ce: Network error: Client connection negotiation failed: client connection to 127.1.226.193:37941: connect: Connection refused (error 111)
I20260928 04:47:04.572352 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 19655
I20260928 04:47:04.577636 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 19786
I20260928 04:47:04.582403 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 19433
2026-09-28T04:47:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (704 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:47:04 dist-test-slave-j5c3 krb5kdc[19957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:47:04 dist-test-slave-j5c3 krb5kdc[19957](info): set up 2 sockets
Sep 28 04:47:04 dist-test-slave-j5c3 krb5kdc[19957](info): commencing operation
krb5kdc: starting...
W20260928 04:47:05.632040 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:47:05 dist-test-slave-j5c3 krb5kdc[19957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:47:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:47:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:47:05.758860 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:38749
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:41909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:38749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:05.842667 19970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:05.842934 19970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:05.842974 19970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:05.844280 19970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:05.844338 19970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:05.844363 19970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:05.844393 19970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:05.844419 19970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:05.845930 19970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:41909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:38749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:38749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:05.846204 19970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:05.846432 19970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:05.851318 19975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:05.851404 19978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:05.851495 19970 server_base.cc:1034] running on GCE node
W20260928 04:47:05.851284 19976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:05.851837 19970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:05.852124 19970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:05.853312 19970 hybrid_clock.cc:648] HybridClock initialized: now 1790570825853277 us; error 50 us; skew 500 ppm
Sep 28 04:47:05 dist-test-slave-j5c3 krb5kdc[19957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570825, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:05.855844 19970 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:47:05.856400 19970 webserver.cc:460] Webserver started at http://127.1.226.254:42417/ using document root <none> and password file <none>
I20260928 04:47:05.856654 19970 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:05.856727 19970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:05.856877 19970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:05.857981 19970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "de4e9ece07ba4d6b82421e67475f5491"
format_stamp: "Formatted at 2026-09-28 04:47:05 on dist-test-slave-j5c3"
server_key: "c414f5a92393c0b7cee98e55b840aeee"
server_key_iv: "aeaa2feb818105dbcaa0fd6e1ab8fb43"
server_key_version: "encryptionkey@0"
I20260928 04:47:05.858352 19970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "de4e9ece07ba4d6b82421e67475f5491"
format_stamp: "Formatted at 2026-09-28 04:47:05 on dist-test-slave-j5c3"
server_key: "c414f5a92393c0b7cee98e55b840aeee"
server_key_iv: "aeaa2feb818105dbcaa0fd6e1ab8fb43"
server_key_version: "encryptionkey@0"
I20260928 04:47:05.859709 19970 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 04:47:05.860558 19985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:05.860750 19970 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:05.860865 19970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "de4e9ece07ba4d6b82421e67475f5491"
format_stamp: "Formatted at 2026-09-28 04:47:05 on dist-test-slave-j5c3"
server_key: "c414f5a92393c0b7cee98e55b840aeee"
server_key_iv: "aeaa2feb818105dbcaa0fd6e1ab8fb43"
server_key_version: "encryptionkey@0"
I20260928 04:47:05.860944 19970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:05.871475 19970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:05.873935 19970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:05.874133 19970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:05.878767 19970 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:38749
I20260928 04:47:05.878810 20037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:38749 every 8 connection(s)
I20260928 04:47:05.879201 19970 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:47:05.880105 20038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:05.882471 20038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de4e9ece07ba4d6b82421e67475f5491: Bootstrap starting.
I20260928 04:47:05.883203 20038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de4e9ece07ba4d6b82421e67475f5491: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:05.883471 20038 log.cc:824] T 00000000000000000000000000000000 P de4e9ece07ba4d6b82421e67475f5491: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:05.883886 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 19970
I20260928 04:47:05.884044 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:47:05.884295 1931 external_mini_cluster.cc:1428] Setting key ee3edf8309b9ea9de4c3a47f926a84c4
I20260928 04:47:05.884460 20038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de4e9ece07ba4d6b82421e67475f5491: No bootstrap required, opened a new log
I20260928 04:47:05.886652 20038 raft_consensus.cc:348] T 00000000000000000000000000000000 P de4e9ece07ba4d6b82421e67475f5491 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4e9ece07ba4d6b82421e67475f5491" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38749 } }
I20260928 04:47:05.886803 20038 raft_consensus.cc:374] T 00000000000000000000000000000000 P de4e9ece07ba4d6b82421e67475f5491 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:05.886866 20038 raft_consensus.cc:729] T 00000000000000000000000000000000 P de4e9ece07ba4d6b82421e67475f5491 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de4e9ece07ba4d6b82421e67475f5491, State: Initialized, Role: FOLLOWER
I20260928 04:47:05.887004 20038 consensus_queue.cc:260] T 00000000000000000000000000000000 P de4e9ece07ba4d6b82421e67475f5491 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4e9ece07ba4d6b82421e67475f5491" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38749 } }
I20260928 04:47:05.887087 20038 raft_consensus.cc:388] T 00000000000000000000000000000000 P de4e9ece07ba4d6b82421e67475f5491 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:47:05.887113 20038 raft_consensus.cc:482] T 00000000000000000000000000000000 P de4e9ece07ba4d6b82421e67475f5491 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:47:05.887185 20038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de4e9ece07ba4d6b82421e67475f5491 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:05.887846 20038 raft_consensus.cc:504] T 00000000000000000000000000000000 P de4e9ece07ba4d6b82421e67475f5491 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4e9ece07ba4d6b82421e67475f5491" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38749 } }
I20260928 04:47:05.887938 20038 leader_election.cc:302] T 00000000000000000000000000000000 P de4e9ece07ba4d6b82421e67475f5491 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de4e9ece07ba4d6b82421e67475f5491; no voters:
I20260928 04:47:05.888196 20038 leader_election.cc:288] T 00000000000000000000000000000000 P de4e9ece07ba4d6b82421e67475f5491 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:47:05.888214 20043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de4e9ece07ba4d6b82421e67475f5491 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:05.888363 20043 raft_consensus.cc:686] T 00000000000000000000000000000000 P de4e9ece07ba4d6b82421e67475f5491 [term 1 LEADER]: Becoming Leader. State: Replica: de4e9ece07ba4d6b82421e67475f5491, State: Running, Role: LEADER
I20260928 04:47:05.888491 20043 consensus_queue.cc:237] T 00000000000000000000000000000000 P de4e9ece07ba4d6b82421e67475f5491 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4e9ece07ba4d6b82421e67475f5491" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38749 } }
I20260928 04:47:05.888535 20038 sys_catalog.cc:564] T 00000000000000000000000000000000 P de4e9ece07ba4d6b82421e67475f5491 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:05.888955 20045 sys_catalog.cc:455] T 00000000000000000000000000000000 P de4e9ece07ba4d6b82421e67475f5491 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de4e9ece07ba4d6b82421e67475f5491" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4e9ece07ba4d6b82421e67475f5491" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38749 } } }
I20260928 04:47:05.889034 20045 sys_catalog.cc:458] T 00000000000000000000000000000000 P de4e9ece07ba4d6b82421e67475f5491 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:05.889549 20044 sys_catalog.cc:455] T 00000000000000000000000000000000 P de4e9ece07ba4d6b82421e67475f5491 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de4e9ece07ba4d6b82421e67475f5491. Latest consensus state: current_term: 1 leader_uuid: "de4e9ece07ba4d6b82421e67475f5491" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4e9ece07ba4d6b82421e67475f5491" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38749 } } }
I20260928 04:47:05.889626 20044 sys_catalog.cc:458] T 00000000000000000000000000000000 P de4e9ece07ba4d6b82421e67475f5491 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:05.889755 20048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:47:05.890407 20048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 04:47:05 dist-test-slave-j5c3 krb5kdc[19957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:47:05.892935 20048 catalog_manager.cc:1347] Generated new cluster ID: 6c19335794044473bb9ed7427ca3b968
I20260928 04:47:05.892992 20048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:47:05.896770 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:05.884914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51308 (local address 127.1.226.254:38749)
0928 04:47:05.885140 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:05.885148 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:05.885173 (+ 25us) server_negotiation.cc:407] Connection header received
0928 04:47:05.885912 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:05.885919 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:05.885969 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:05.886119 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:05.886762 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:05.888187 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:05.889332 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:05.889992 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:05.890080 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:05.893352 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:05.893356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:05.893356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:05.893368 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:05.895087 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:05.895458 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:05.895462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:05.895462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:05.895552 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:05.895769 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:05.895770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:05.895770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:05.896001 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:05.896187 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:05.896335 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 04:47:05.896480 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":125,"threads_started":1}
I20260928 04:47:05.925130 20048 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:47:05.925972 20048 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:47:05.930217 20048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de4e9ece07ba4d6b82421e67475f5491: Generated new TSK 0
I20260928 04:47:05.930438 20048 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.226.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
I20260928 04:47:05.961228 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:0
--local_ip_for_outbound_sockets=127.1.226.193
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:38749
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:41909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:06.045967 20064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:06.046224 20064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:06.046267 20064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:06.047467 20064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:06.047536 20064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:06.047585 20064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:47:06.049141 20064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:41909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:38749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:06.049429 20064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:06.049669 20064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:06.054749 20069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:06.054764 20070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:06.054764 20072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:06.055040 20064 server_base.cc:1034] running on GCE node
I20260928 04:47:06.055241 20064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:06.055470 20064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:06.056618 20064 hybrid_clock.cc:648] HybridClock initialized: now 1790570826056601 us; error 42 us; skew 500 ppm
Sep 28 04:47:06 dist-test-slave-j5c3 krb5kdc[19957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570826, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:06.059154 20064 init.cc:377] Logged in from keytab as kudu/127.1.226.193@KRBTEST.COM (short username kudu)
I20260928 04:47:06.059533 20064 webserver.cc:460] Webserver started at http://127.1.226.193:38661/ using document root <none> and password file <none>
I20260928 04:47:06.059719 20064 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:06.059758 20064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:06.059845 20064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:06.060905 20064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-0/data/instance:
uuid: "63e2b1f1a9704315a28eadbfa2469c68"
format_stamp: "Formatted at 2026-09-28 04:47:06 on dist-test-slave-j5c3"
server_key: "7868bb02958dae32f66b70c92fb76b34"
server_key_iv: "d8540fcc76ed4839e6ca429036947e43"
server_key_version: "encryptionkey@0"
I20260928 04:47:06.061200 20064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance:
uuid: "63e2b1f1a9704315a28eadbfa2469c68"
format_stamp: "Formatted at 2026-09-28 04:47:06 on dist-test-slave-j5c3"
server_key: "7868bb02958dae32f66b70c92fb76b34"
server_key_iv: "d8540fcc76ed4839e6ca429036947e43"
server_key_version: "encryptionkey@0"
I20260928 04:47:06.062469 20064 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:47:06.063311 20079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:06.063520 20064 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:06.063633 20064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "63e2b1f1a9704315a28eadbfa2469c68"
format_stamp: "Formatted at 2026-09-28 04:47:06 on dist-test-slave-j5c3"
server_key: "7868bb02958dae32f66b70c92fb76b34"
server_key_iv: "d8540fcc76ed4839e6ca429036947e43"
server_key_version: "encryptionkey@0"
I20260928 04:47:06.063721 20064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:06.074771 20064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:06.077193 20064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:06.077397 20064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:06.077661 20064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:47:06.077991 20064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:47:06.078025 20064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:06.078048 20064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:47:06.078083 20064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:06.083777 20064 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:38689
I20260928 04:47:06.083906 20192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:38689 every 8 connection(s)
I20260928 04:47:06.084244 20064 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
I20260928 04:47:06.086877 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 20064
I20260928 04:47:06.087010 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance
I20260928 04:47:06.087211 1931 external_mini_cluster.cc:1428] Setting key 52429128bfa78418dc415ae3059d411e
Sep 28 04:47:06 dist-test-slave-j5c3 krb5kdc[19957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570826, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:47:06.096777 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:06.085164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:46747 (local address 127.1.226.254:38749)
0928 04:47:06.085243 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:06.085245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:06.085422 (+ 177us) server_negotiation.cc:407] Connection header received
0928 04:47:06.085930 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:06.085931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:06.085944 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:06.086019 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:06.087242 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.087988 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.089744 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.090130 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.090214 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:06.093153 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:06.093156 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:06.093156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:06.093166 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:06.095114 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:06.095566 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:06.095568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:06.095569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:06.095638 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:06.095949 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:06.095949 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:06.095950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:06.096111 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:06.096204 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:06.096509 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 04:47:06.096615 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 04:47:06.096872 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:06.085083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38749 (local address 127.1.226.193:46747)
0928 04:47:06.085282 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:06.085327 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:06.085813 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:06.086099 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:06.086103 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:06.086773 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:06.087097 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:06.087105 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.088102 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.088102 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:06.089614 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:06.089618 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.090235 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.090236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:06.090368 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:06.091163 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:47:06.091175 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:06.093038 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:06.095223 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:06.095231 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:06.095233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:06.095481 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:06.095743 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:06.095744 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:06.095744 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:06.095868 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:06.096216 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:06.096223 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:06.096370 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 04:47:06.096462 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 04:47:06.096601 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for kudu/127.1.226.194@KRBTEST.COM; defaulting to no policy
I20260928 04:47:06.097393 20193 heartbeater.cc:347] Connected to a master server at 127.1.226.254:38749
I20260928 04:47:06.097513 20193 heartbeater.cc:464] Registering TS with master...
I20260928 04:47:06.097709 20193 heartbeater.cc:511] Master 127.1.226.254:38749 requested a full tablet report, sending...
I20260928 04:47:06.098241 20002 ts_manager.cc:194] Registered new tserver with Master: 63e2b1f1a9704315a28eadbfa2469c68 (127.1.226.193:38689)
I20260928 04:47:06.098946 20002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.193@KRBTEST.COM'} at 127.1.226.193:46747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
I20260928 04:47:06.115382 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:0
--local_ip_for_outbound_sockets=127.1.226.194
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:38749
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:41909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:06.199424 20198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:06.199647 20198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:06.199673 20198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:06.200850 20198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:06.200887 20198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:06.200912 20198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:47:06.202303 20198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:41909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:38749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:06.202497 20198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:06.202688 20198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:06.207998 20203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:06.208247 20198 server_base.cc:1034] running on GCE node
W20260928 04:47:06.208006 20206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:06.208010 20204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:06.210459 20198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:06.210727 20198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:06.211894 20198 hybrid_clock.cc:648] HybridClock initialized: now 1790570826211884 us; error 48 us; skew 500 ppm
Sep 28 04:47:06 dist-test-slave-j5c3 krb5kdc[19957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570826, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:06.214740 20198 init.cc:377] Logged in from keytab as kudu/127.1.226.194@KRBTEST.COM (short username kudu)
I20260928 04:47:06.215202 20198 webserver.cc:460] Webserver started at http://127.1.226.194:41771/ using document root <none> and password file <none>
I20260928 04:47:06.215458 20198 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:06.215529 20198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:06.215673 20198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:06.216850 20198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-1/data/instance:
uuid: "b94893b7379e4b098a7161653a42857e"
format_stamp: "Formatted at 2026-09-28 04:47:06 on dist-test-slave-j5c3"
server_key: "02dc4848ba45e1d4ca0e79345cb580ec"
server_key_iv: "eb82e3c39fc2a05aa7c37284f8ad9826"
server_key_version: "encryptionkey@0"
I20260928 04:47:06.217214 20198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance:
uuid: "b94893b7379e4b098a7161653a42857e"
format_stamp: "Formatted at 2026-09-28 04:47:06 on dist-test-slave-j5c3"
server_key: "02dc4848ba45e1d4ca0e79345cb580ec"
server_key_iv: "eb82e3c39fc2a05aa7c37284f8ad9826"
server_key_version: "encryptionkey@0"
I20260928 04:47:06.218552 20198 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:47:06.219328 20213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:06.219547 20198 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:06.219640 20198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "b94893b7379e4b098a7161653a42857e"
format_stamp: "Formatted at 2026-09-28 04:47:06 on dist-test-slave-j5c3"
server_key: "02dc4848ba45e1d4ca0e79345cb580ec"
server_key_iv: "eb82e3c39fc2a05aa7c37284f8ad9826"
server_key_version: "encryptionkey@0"
I20260928 04:47:06.219739 20198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:06.243045 20198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:06.245380 20198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:06.245558 20198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:06.245862 20198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:47:06.246206 20198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:47:06.246263 20198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:06.246304 20198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:47:06.246333 20198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:06.252295 20198 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:43851
I20260928 04:47:06.252373 20326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:43851 every 8 connection(s)
I20260928 04:47:06.252704 20198 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
Sep 28 04:47:06 dist-test-slave-j5c3 krb5kdc[19957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570826, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:47:06.260968 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 20198
I20260928 04:47:06.261098 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance
I20260928 04:47:06.261332 1931 external_mini_cluster.cc:1428] Setting key 28f66262906fcbfee024531e769faac6
I20260928 04:47:06.262560 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:06.253457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:33029 (local address 127.1.226.254:38749)
0928 04:47:06.253517 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:06.253519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:06.253732 (+ 213us) server_negotiation.cc:407] Connection header received
0928 04:47:06.254283 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:06.254284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:06.254305 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:06.254367 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:06.255335 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.255840 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.256918 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.257162 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.257243 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:06.259500 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:06.259502 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:06.259502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:06.259509 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:06.260969 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:06.261453 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:06.261454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:06.261454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:06.261536 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:06.261845 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:06.261846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:06.261846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:06.261986 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:06.262058 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:06.262428 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 04:47:06.262495 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 04:47:06.262849 20329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:06.253444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38749 (local address 127.1.226.194:33029)
0928 04:47:06.253615 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:06.253652 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:06.254179 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:06.254439 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:06.254444 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:06.255025 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:06.255216 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:06.255220 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.255945 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.255946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:06.256812 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:06.256814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.257252 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.257253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:06.257388 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:06.257917 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:47:06.257925 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:06.259410 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:06.261079 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:06.261086 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:06.261089 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:06.261384 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:06.261633 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:06.261634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:06.261634 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:06.261775 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:06.262066 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:06.262073 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:06.262286 (+ 213us) client_negotiation.cc:769] Sending connection context
0928 04:47:06.262408 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 04:47:06.262592 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
I20260928 04:47:06.263373 20327 heartbeater.cc:347] Connected to a master server at 127.1.226.254:38749
I20260928 04:47:06.263463 20327 heartbeater.cc:464] Registering TS with master...
I20260928 04:47:06.263681 20327 heartbeater.cc:511] Master 127.1.226.254:38749 requested a full tablet report, sending...
I20260928 04:47:06.264187 20002 ts_manager.cc:194] Registered new tserver with Master: b94893b7379e4b098a7161653a42857e (127.1.226.194:43851)
I20260928 04:47:06.264701 20002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.194@KRBTEST.COM'} at 127.1.226.194:33029
WARNING: no policy specified for kudu/127.1.226.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
I20260928 04:47:06.286667 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:0
--local_ip_for_outbound_sockets=127.1.226.195
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:38749
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:41909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:06.369506 20332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:06.369732 20332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:06.369766 20332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:06.370972 20332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:06.371016 20332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:06.371044 20332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:47:06.372639 20332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:41909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:38749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:06.372900 20332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:06.373139 20332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:06.378249 20338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:06.378305 20337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:06.378261 20340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:06.378504 20332 server_base.cc:1034] running on GCE node
I20260928 04:47:06.378696 20332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:06.378975 20332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:06.380213 20332 hybrid_clock.cc:648] HybridClock initialized: now 1790570826380141 us; error 43 us; skew 500 ppm
Sep 28 04:47:06 dist-test-slave-j5c3 krb5kdc[19957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570826, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:06.382784 20332 init.cc:377] Logged in from keytab as kudu/127.1.226.195@KRBTEST.COM (short username kudu)
I20260928 04:47:06.383248 20332 webserver.cc:460] Webserver started at http://127.1.226.195:33983/ using document root <none> and password file <none>
I20260928 04:47:06.383472 20332 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:06.383541 20332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:06.383690 20332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:06.384869 20332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-2/data/instance:
uuid: "f5624cd530ae4f55add98837eb0efad7"
format_stamp: "Formatted at 2026-09-28 04:47:06 on dist-test-slave-j5c3"
server_key: "8a02d2c5647047ba53f1a34a66411520"
server_key_iv: "ec82eb6c78e52f8976109ed03818571c"
server_key_version: "encryptionkey@0"
I20260928 04:47:06.385243 20332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance:
uuid: "f5624cd530ae4f55add98837eb0efad7"
format_stamp: "Formatted at 2026-09-28 04:47:06 on dist-test-slave-j5c3"
server_key: "8a02d2c5647047ba53f1a34a66411520"
server_key_iv: "ec82eb6c78e52f8976109ed03818571c"
server_key_version: "encryptionkey@0"
I20260928 04:47:06.386547 20332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:47:06.387395 20347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:06.387619 20332 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:06.387722 20332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "f5624cd530ae4f55add98837eb0efad7"
format_stamp: "Formatted at 2026-09-28 04:47:06 on dist-test-slave-j5c3"
server_key: "8a02d2c5647047ba53f1a34a66411520"
server_key_iv: "ec82eb6c78e52f8976109ed03818571c"
server_key_version: "encryptionkey@0"
I20260928 04:47:06.387821 20332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:06.402899 20332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:06.405090 20332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:06.405267 20332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:06.405519 20332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:47:06.405838 20332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:47:06.405893 20332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:06.405936 20332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:47:06.405962 20332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:06.412079 20332 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:36203
I20260928 04:47:06.412122 20460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:36203 every 8 connection(s)
I20260928 04:47:06.412518 20332 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
Sep 28 04:47:06 dist-test-slave-j5c3 krb5kdc[19957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570826, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:47:06.422122 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:06.413311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:34423 (local address 127.1.226.254:38749)
0928 04:47:06.413378 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:06.413380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:06.413636 (+ 256us) server_negotiation.cc:407] Connection header received
0928 04:47:06.414111 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:06.414113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:06.414123 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:06.414181 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:06.415137 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.415716 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.416872 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.417067 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.417134 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:06.419388 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:06.419389 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:06.419389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:06.419396 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:06.420873 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:06.421248 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:06.421248 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:06.421249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:06.421298 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:06.421520 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:06.421520 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:06.421520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:06.421621 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:06.421670 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:06.421920 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 04:47:06.421981 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 04:47:06.422194 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 20332
I20260928 04:47:06.422217 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:06.413338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:38749 (local address 127.1.226.195:34423)
0928 04:47:06.413506 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:06.413544 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:06.414022 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:06.414256 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:06.414259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:06.414833 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:06.415028 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:06.415033 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.415831 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.415833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:06.416779 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:06.416782 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.417150 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.417151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:06.417266 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:06.417775 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:47:06.417825 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:06.419267 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:06.420949 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:06.420955 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:06.420957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:06.421181 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:06.421359 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:06.421360 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:06.421360 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:06.421463 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:06.421677 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:06.421681 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:06.421808 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 04:47:06.421892 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 04:47:06.422017 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260928 04:47:06.422278 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance
I20260928 04:47:06.422483 1931 external_mini_cluster.cc:1428] Setting key a028f8ef4e5a6d9079db89604c6b3f0a
I20260928 04:47:06.422821 20461 heartbeater.cc:347] Connected to a master server at 127.1.226.254:38749
I20260928 04:47:06.422925 20461 heartbeater.cc:464] Registering TS with master...
I20260928 04:47:06.423122 20461 heartbeater.cc:511] Master 127.1.226.254:38749 requested a full tablet report, sending...
I20260928 04:47:06.423534 20002 ts_manager.cc:194] Registered new tserver with Master: f5624cd530ae4f55add98837eb0efad7 (127.1.226.195:36203)
I20260928 04:47:06.423935 20002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.195@KRBTEST.COM'} at 127.1.226.195:34423
I20260928 04:47:06.434448 1931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:47:06.442430 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:06.435870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51324 (local address 127.1.226.254:38749)
0928 04:47:06.435931 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:06.435933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:06.435992 (+ 59us) server_negotiation.cc:407] Connection header received
0928 04:47:06.436032 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:06.436033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:06.436042 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:06.436102 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:06.436721 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.437235 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.437936 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.438112 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.438165 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:06.439619 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:06.439620 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:06.439620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:06.439625 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:06.441085 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:06.441540 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:06.441541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:06.441542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:06.441601 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:06.441892 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:06.441893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:06.441894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:06.442034 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:06.442121 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:06.442306 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 04:47:06.442363 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 04:47:06.444291 20002 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:47:06.445935 20002 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:47:06.452426 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:06.448542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:36203 (local address 127.0.0.1:34480)
0928 04:47:06.448662 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:06.448674 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:06.448727 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:06.449143 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:06.449143 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:06.449145 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:06.449182 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:06.449183 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.450486 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.450487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:06.451361 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:06.451363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.452116 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.452117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:06.452224 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:06.452243 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 04:47:06.452292 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 04:47:06.452343 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
I20260928 04:47:06.452524 20476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:06.448354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:38689 (local address 127.0.0.1:44736)
0928 04:47:06.448657 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:06.448666 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:06.448706 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:06.449031 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:06.449035 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:06.449038 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:06.449122 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:06.449125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.450160 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.450161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:06.451330 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:06.451333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.452249 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.452250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:06.452310 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:06.452314 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:47:06.452374 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 04:47:06.452416 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":81,"threads_started":1}
I20260928 04:47:06.452684 20474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:06.448489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44736 (local address 127.1.226.193:38689)
0928 04:47:06.448865 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:06.448870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:06.448880 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:47:06.448922 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:06.448924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:06.448940 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:06.449021 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:06.449206 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.450051 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.451429 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.452120 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.452246 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:06.452320 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:06.452389 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 04:47:06.452429 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":80,"threads_started":1}
I20260928 04:47:06.453013 20477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:06.448738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34480 (local address 127.1.226.195:36203)
0928 04:47:06.448990 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:06.448994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:06.449003 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:47:06.449040 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:06.449043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:06.449057 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:06.449143 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:06.449249 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.450356 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.451458 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.452020 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.452793 (+ 773us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:06.452859 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:06.452885 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 04:47:06.452922 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":152,"threads_started":1}
I20260928 04:47:06.453087 20473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:06.448322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:43851 (local address 127.0.0.1:44580)
0928 04:47:06.448529 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:06.448574 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:06.448643 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:06.449091 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:06.449092 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:06.449095 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:06.449347 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:06.449349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.450653 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.450654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:06.451851 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:06.451854 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.452882 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.452883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:06.452941 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:06.452944 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:47:06.452973 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 04:47:06.453011 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":66,"threads_started":1}
I20260928 04:47:06.453372 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:06.448471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44580 (local address 127.1.226.194:43851)
0928 04:47:06.448802 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:06.448807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:06.448817 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:47:06.448860 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:06.448862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:06.448874 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:06.449002 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:06.449545 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.450552 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.451948 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.452789 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.453183 (+ 394us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:06.453254 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:06.453285 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 04:47:06.453316 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":101,"threads_started":1}
I20260928 04:47:06.453430 20127 tablet_service.cc:1467] Processing CreateTablet for tablet d2c80a28aa9a47e680bee6d5647a3f16 (DEFAULT_TABLE table=test-table [id=15086e8634724b3c9c9c6ce2e9b2c760]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:47:06.453838 20127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2c80a28aa9a47e680bee6d5647a3f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:06.454062 20261 tablet_service.cc:1467] Processing CreateTablet for tablet d2c80a28aa9a47e680bee6d5647a3f16 (DEFAULT_TABLE table=test-table [id=15086e8634724b3c9c9c6ce2e9b2c760]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:47:06.454360 20261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2c80a28aa9a47e680bee6d5647a3f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:06.456718 20478 tablet_bootstrap.cc:492] T d2c80a28aa9a47e680bee6d5647a3f16 P 63e2b1f1a9704315a28eadbfa2469c68: Bootstrap starting.
I20260928 04:47:06.457134 20395 tablet_service.cc:1467] Processing CreateTablet for tablet d2c80a28aa9a47e680bee6d5647a3f16 (DEFAULT_TABLE table=test-table [id=15086e8634724b3c9c9c6ce2e9b2c760]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:47:06.457446 20395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2c80a28aa9a47e680bee6d5647a3f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:06.457454 20478 tablet_bootstrap.cc:654] T d2c80a28aa9a47e680bee6d5647a3f16 P 63e2b1f1a9704315a28eadbfa2469c68: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:06.457932 20478 log.cc:824] T d2c80a28aa9a47e680bee6d5647a3f16 P 63e2b1f1a9704315a28eadbfa2469c68: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:06.458254 20479 tablet_bootstrap.cc:492] T d2c80a28aa9a47e680bee6d5647a3f16 P b94893b7379e4b098a7161653a42857e: Bootstrap starting.
I20260928 04:47:06.458753 20478 tablet_bootstrap.cc:492] T d2c80a28aa9a47e680bee6d5647a3f16 P 63e2b1f1a9704315a28eadbfa2469c68: No bootstrap required, opened a new log
I20260928 04:47:06.458825 20478 ts_tablet_manager.cc:1397] T d2c80a28aa9a47e680bee6d5647a3f16 P 63e2b1f1a9704315a28eadbfa2469c68: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:47:06.458889 20479 tablet_bootstrap.cc:654] T d2c80a28aa9a47e680bee6d5647a3f16 P b94893b7379e4b098a7161653a42857e: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:06.459434 20479 log.cc:824] T d2c80a28aa9a47e680bee6d5647a3f16 P b94893b7379e4b098a7161653a42857e: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:06.460332 20479 tablet_bootstrap.cc:492] T d2c80a28aa9a47e680bee6d5647a3f16 P b94893b7379e4b098a7161653a42857e: No bootstrap required, opened a new log
I20260928 04:47:06.460390 20479 ts_tablet_manager.cc:1397] T d2c80a28aa9a47e680bee6d5647a3f16 P b94893b7379e4b098a7161653a42857e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:47:06.460951 20478 raft_consensus.cc:348] T d2c80a28aa9a47e680bee6d5647a3f16 P 63e2b1f1a9704315a28eadbfa2469c68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94893b7379e4b098a7161653a42857e" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43851 } } peers { permanent_uuid: "f5624cd530ae4f55add98837eb0efad7" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 36203 } } peers { permanent_uuid: "63e2b1f1a9704315a28eadbfa2469c68" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 38689 } }
I20260928 04:47:06.461129 20478 raft_consensus.cc:374] T d2c80a28aa9a47e680bee6d5647a3f16 P 63e2b1f1a9704315a28eadbfa2469c68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:06.461163 20478 raft_consensus.cc:729] T d2c80a28aa9a47e680bee6d5647a3f16 P 63e2b1f1a9704315a28eadbfa2469c68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63e2b1f1a9704315a28eadbfa2469c68, State: Initialized, Role: FOLLOWER
I20260928 04:47:06.461278 20478 consensus_queue.cc:260] T d2c80a28aa9a47e680bee6d5647a3f16 P 63e2b1f1a9704315a28eadbfa2469c68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94893b7379e4b098a7161653a42857e" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43851 } } peers { permanent_uuid: "f5624cd530ae4f55add98837eb0efad7" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 36203 } } peers { permanent_uuid: "63e2b1f1a9704315a28eadbfa2469c68" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 38689 } }
I20260928 04:47:06.461556 20478 ts_tablet_manager.cc:1428] T d2c80a28aa9a47e680bee6d5647a3f16 P 63e2b1f1a9704315a28eadbfa2469c68: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:47:06.461707 20193 heartbeater.cc:503] Master 127.1.226.254:38749 was elected leader, sending a full tablet report...
I20260928 04:47:06.462366 20479 raft_consensus.cc:348] T d2c80a28aa9a47e680bee6d5647a3f16 P b94893b7379e4b098a7161653a42857e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94893b7379e4b098a7161653a42857e" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43851 } } peers { permanent_uuid: "f5624cd530ae4f55add98837eb0efad7" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 36203 } } peers { permanent_uuid: "63e2b1f1a9704315a28eadbfa2469c68" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 38689 } }
I20260928 04:47:06.462517 20479 raft_consensus.cc:374] T d2c80a28aa9a47e680bee6d5647a3f16 P b94893b7379e4b098a7161653a42857e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:06.462559 20479 raft_consensus.cc:729] T d2c80a28aa9a47e680bee6d5647a3f16 P b94893b7379e4b098a7161653a42857e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94893b7379e4b098a7161653a42857e, State: Initialized, Role: FOLLOWER
I20260928 04:47:06.462667 20479 consensus_queue.cc:260] T d2c80a28aa9a47e680bee6d5647a3f16 P b94893b7379e4b098a7161653a42857e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94893b7379e4b098a7161653a42857e" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43851 } } peers { permanent_uuid: "f5624cd530ae4f55add98837eb0efad7" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 36203 } } peers { permanent_uuid: "63e2b1f1a9704315a28eadbfa2469c68" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 38689 } }
I20260928 04:47:06.462927 20479 ts_tablet_manager.cc:1428] T d2c80a28aa9a47e680bee6d5647a3f16 P b94893b7379e4b098a7161653a42857e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:47:06.462994 20327 heartbeater.cc:503] Master 127.1.226.254:38749 was elected leader, sending a full tablet report...
I20260928 04:47:06.463136 20482 tablet_bootstrap.cc:492] T d2c80a28aa9a47e680bee6d5647a3f16 P f5624cd530ae4f55add98837eb0efad7: Bootstrap starting.
I20260928 04:47:06.463847 20482 tablet_bootstrap.cc:654] T d2c80a28aa9a47e680bee6d5647a3f16 P f5624cd530ae4f55add98837eb0efad7: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:06.464409 20482 log.cc:824] T d2c80a28aa9a47e680bee6d5647a3f16 P f5624cd530ae4f55add98837eb0efad7: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:06.465094 20482 tablet_bootstrap.cc:492] T d2c80a28aa9a47e680bee6d5647a3f16 P f5624cd530ae4f55add98837eb0efad7: No bootstrap required, opened a new log
I20260928 04:47:06.465147 20482 ts_tablet_manager.cc:1397] T d2c80a28aa9a47e680bee6d5647a3f16 P f5624cd530ae4f55add98837eb0efad7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:47:06.466533 20482 raft_consensus.cc:348] T d2c80a28aa9a47e680bee6d5647a3f16 P f5624cd530ae4f55add98837eb0efad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94893b7379e4b098a7161653a42857e" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43851 } } peers { permanent_uuid: "f5624cd530ae4f55add98837eb0efad7" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 36203 } } peers { permanent_uuid: "63e2b1f1a9704315a28eadbfa2469c68" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 38689 } }
I20260928 04:47:06.466672 20482 raft_consensus.cc:374] T d2c80a28aa9a47e680bee6d5647a3f16 P f5624cd530ae4f55add98837eb0efad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:06.466694 20482 raft_consensus.cc:729] T d2c80a28aa9a47e680bee6d5647a3f16 P f5624cd530ae4f55add98837eb0efad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5624cd530ae4f55add98837eb0efad7, State: Initialized, Role: FOLLOWER
I20260928 04:47:06.466791 20482 consensus_queue.cc:260] T d2c80a28aa9a47e680bee6d5647a3f16 P f5624cd530ae4f55add98837eb0efad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94893b7379e4b098a7161653a42857e" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43851 } } peers { permanent_uuid: "f5624cd530ae4f55add98837eb0efad7" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 36203 } } peers { permanent_uuid: "63e2b1f1a9704315a28eadbfa2469c68" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 38689 } }
I20260928 04:47:06.466995 20482 ts_tablet_manager.cc:1428] T d2c80a28aa9a47e680bee6d5647a3f16 P f5624cd530ae4f55add98837eb0efad7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:47:06.467069 20461 heartbeater.cc:503] Master 127.1.226.254:38749 was elected leader, sending a full tablet report...
W20260928 04:47:06.503309 20328 tablet.cc:2369] T d2c80a28aa9a47e680bee6d5647a3f16 P b94893b7379e4b098a7161653a42857e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:47:06.524861 20483 raft_consensus.cc:482] T d2c80a28aa9a47e680bee6d5647a3f16 P 63e2b1f1a9704315a28eadbfa2469c68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:47:06.524982 20483 raft_consensus.cc:504] T d2c80a28aa9a47e680bee6d5647a3f16 P 63e2b1f1a9704315a28eadbfa2469c68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94893b7379e4b098a7161653a42857e" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43851 } } peers { permanent_uuid: "f5624cd530ae4f55add98837eb0efad7" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 36203 } } peers { permanent_uuid: "63e2b1f1a9704315a28eadbfa2469c68" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 38689 } }
I20260928 04:47:06.525382 20483 leader_election.cc:288] T d2c80a28aa9a47e680bee6d5647a3f16 P 63e2b1f1a9704315a28eadbfa2469c68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b94893b7379e4b098a7161653a42857e (127.1.226.194:43851), f5624cd530ae4f55add98837eb0efad7 (127.1.226.195:36203)
I20260928 04:47:06.528792 20477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:06.525925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:54549 (local address 127.1.226.195:36203)
0928 04:47:06.525991 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:06.525993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:06.526108 (+ 115us) server_negotiation.cc:407] Connection header received
0928 04:47:06.526143 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:06.526144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:06.526154 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:06.526208 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:06.526384 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.526825 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.527954 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.528411 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.528469 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:06.528496 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:06.528672 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 04:47:06.528714 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 04:47:06.528815 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:06.525668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:43851 (local address 127.1.226.193:32853)
0928 04:47:06.525753 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:06.525783 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:06.525841 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:06.526052 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:06.526053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:06.526055 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:06.526148 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:06.526150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.527020 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.527021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:06.527862 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:06.527865 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.528526 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.528527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:06.528610 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:06.528612 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:47:06.528687 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 04:47:06.528724 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":71}
I20260928 04:47:06.528825 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:06.525746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:32853 (local address 127.1.226.194:43851)
0928 04:47:06.525861 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:06.525863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:06.525873 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:47:06.525902 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:06.525903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:06.525927 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:06.526002 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:06.526245 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.526918 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.527989 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.528416 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.528517 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:06.528525 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:06.528714 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 04:47:06.528755 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 04:47:06.528826 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:06.525830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:36203 (local address 127.1.226.193:54549)
0928 04:47:06.526019 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:06.526031 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:06.526102 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:06.526239 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:06.526239 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:06.526241 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:06.526315 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:06.526316 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.526918 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.526919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:06.527852 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:06.527854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.528506 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.528507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:06.528591 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:06.528595 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:47:06.528646 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 04:47:06.528715 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":133,"threads_started":1}
I20260928 04:47:06.529178 20281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2c80a28aa9a47e680bee6d5647a3f16" candidate_uuid: "63e2b1f1a9704315a28eadbfa2469c68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94893b7379e4b098a7161653a42857e" is_pre_election: true
I20260928 04:47:06.529356 20281 raft_consensus.cc:2454] T d2c80a28aa9a47e680bee6d5647a3f16 P b94893b7379e4b098a7161653a42857e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63e2b1f1a9704315a28eadbfa2469c68 in term 0.
I20260928 04:47:06.529647 20083 leader_election.cc:302] T d2c80a28aa9a47e680bee6d5647a3f16 P 63e2b1f1a9704315a28eadbfa2469c68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63e2b1f1a9704315a28eadbfa2469c68, b94893b7379e4b098a7161653a42857e; no voters:
I20260928 04:47:06.529649 20415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2c80a28aa9a47e680bee6d5647a3f16" candidate_uuid: "63e2b1f1a9704315a28eadbfa2469c68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5624cd530ae4f55add98837eb0efad7" is_pre_election: true
I20260928 04:47:06.529778 20415 raft_consensus.cc:2454] T d2c80a28aa9a47e680bee6d5647a3f16 P f5624cd530ae4f55add98837eb0efad7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63e2b1f1a9704315a28eadbfa2469c68 in term 0.
I20260928 04:47:06.529814 20483 raft_consensus.cc:2781] T d2c80a28aa9a47e680bee6d5647a3f16 P 63e2b1f1a9704315a28eadbfa2469c68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:47:06.529866 20483 raft_consensus.cc:482] T d2c80a28aa9a47e680bee6d5647a3f16 P 63e2b1f1a9704315a28eadbfa2469c68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:47:06.529893 20483 raft_consensus.cc:3037] T d2c80a28aa9a47e680bee6d5647a3f16 P 63e2b1f1a9704315a28eadbfa2469c68 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:06.530607 20483 raft_consensus.cc:504] T d2c80a28aa9a47e680bee6d5647a3f16 P 63e2b1f1a9704315a28eadbfa2469c68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94893b7379e4b098a7161653a42857e" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43851 } } peers { permanent_uuid: "f5624cd530ae4f55add98837eb0efad7" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 36203 } } peers { permanent_uuid: "63e2b1f1a9704315a28eadbfa2469c68" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 38689 } }
I20260928 04:47:06.530738 20483 leader_election.cc:288] T d2c80a28aa9a47e680bee6d5647a3f16 P 63e2b1f1a9704315a28eadbfa2469c68 [CANDIDATE]: Term 1 election: Requested vote from peers b94893b7379e4b098a7161653a42857e (127.1.226.194:43851), f5624cd530ae4f55add98837eb0efad7 (127.1.226.195:36203)
I20260928 04:47:06.530895 20281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2c80a28aa9a47e680bee6d5647a3f16" candidate_uuid: "63e2b1f1a9704315a28eadbfa2469c68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94893b7379e4b098a7161653a42857e"
I20260928 04:47:06.530978 20281 raft_consensus.cc:3037] T d2c80a28aa9a47e680bee6d5647a3f16 P b94893b7379e4b098a7161653a42857e [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:06.530995 20415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2c80a28aa9a47e680bee6d5647a3f16" candidate_uuid: "63e2b1f1a9704315a28eadbfa2469c68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5624cd530ae4f55add98837eb0efad7"
I20260928 04:47:06.531090 20415 raft_consensus.cc:3037] T d2c80a28aa9a47e680bee6d5647a3f16 P f5624cd530ae4f55add98837eb0efad7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:06.531699 20281 raft_consensus.cc:2454] T d2c80a28aa9a47e680bee6d5647a3f16 P b94893b7379e4b098a7161653a42857e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63e2b1f1a9704315a28eadbfa2469c68 in term 1.
I20260928 04:47:06.531862 20083 leader_election.cc:302] T d2c80a28aa9a47e680bee6d5647a3f16 P 63e2b1f1a9704315a28eadbfa2469c68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63e2b1f1a9704315a28eadbfa2469c68, b94893b7379e4b098a7161653a42857e; no voters:
I20260928 04:47:06.531958 20483 raft_consensus.cc:2781] T d2c80a28aa9a47e680bee6d5647a3f16 P 63e2b1f1a9704315a28eadbfa2469c68 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:06.531996 20415 raft_consensus.cc:2454] T d2c80a28aa9a47e680bee6d5647a3f16 P f5624cd530ae4f55add98837eb0efad7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63e2b1f1a9704315a28eadbfa2469c68 in term 1.
I20260928 04:47:06.532116 20483 raft_consensus.cc:686] T d2c80a28aa9a47e680bee6d5647a3f16 P 63e2b1f1a9704315a28eadbfa2469c68 [term 1 LEADER]: Becoming Leader. State: Replica: 63e2b1f1a9704315a28eadbfa2469c68, State: Running, Role: LEADER
I20260928 04:47:06.532243 20483 consensus_queue.cc:237] T d2c80a28aa9a47e680bee6d5647a3f16 P 63e2b1f1a9704315a28eadbfa2469c68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94893b7379e4b098a7161653a42857e" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43851 } } peers { permanent_uuid: "f5624cd530ae4f55add98837eb0efad7" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 36203 } } peers { permanent_uuid: "63e2b1f1a9704315a28eadbfa2469c68" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 38689 } }
I20260928 04:47:06.533046 20002 catalog_manager.cc:5495] T d2c80a28aa9a47e680bee6d5647a3f16 P 63e2b1f1a9704315a28eadbfa2469c68 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63e2b1f1a9704315a28eadbfa2469c68 (127.1.226.193). New cstate: current_term: 1 leader_uuid: "63e2b1f1a9704315a28eadbfa2469c68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94893b7379e4b098a7161653a42857e" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5624cd530ae4f55add98837eb0efad7" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 36203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63e2b1f1a9704315a28eadbfa2469c68" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 38689 } health_report { overall_health: HEALTHY } } }
I20260928 04:47:06.545105 20474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:06.542820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44750 (local address 127.1.226.193:38689)
0928 04:47:06.542885 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:06.542887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:06.542895 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:47:06.542928 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:06.542929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:06.542946 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:06.543007 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:47:06.543180 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.543650 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.544446 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.544634 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.544690 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:06.544818 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:47:06.544909 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:47:06.544983 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:06.545034 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 04:47:06.545044 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 04:47:06.547104 20415 raft_consensus.cc:1261] T d2c80a28aa9a47e680bee6d5647a3f16 P f5624cd530ae4f55add98837eb0efad7 [term 1 FOLLOWER]: Refusing update from remote peer 63e2b1f1a9704315a28eadbfa2469c68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:47:06.547104 20281 raft_consensus.cc:1261] T d2c80a28aa9a47e680bee6d5647a3f16 P b94893b7379e4b098a7161653a42857e [term 1 FOLLOWER]: Refusing update from remote peer 63e2b1f1a9704315a28eadbfa2469c68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:47:06.547484 20488 consensus_queue.cc:1035] T d2c80a28aa9a47e680bee6d5647a3f16 P 63e2b1f1a9704315a28eadbfa2469c68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5624cd530ae4f55add98837eb0efad7" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 36203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:47:06.547616 20483 consensus_queue.cc:1035] T d2c80a28aa9a47e680bee6d5647a3f16 P 63e2b1f1a9704315a28eadbfa2469c68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b94893b7379e4b098a7161653a42857e" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:47:06.550810 20492 mvcc.cc:205] Tried to move back new op lower bound from 7334178105533595648 to 7334178105476587520. Current Snapshot: MvccSnapshot[applied={T|T < 7334178105533595648 or (T in {7334178105533595648})}]
I20260928 04:47:06.553167 20001 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:47:06.553279 20001 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:47:06.554102 20001 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de4e9ece07ba4d6b82421e67475f5491: Sending DeleteTablet for 3 replicas of tablet d2c80a28aa9a47e680bee6d5647a3f16
I20260928 04:47:06.554455 20127 tablet_service.cc:1514] Processing DeleteTablet for tablet d2c80a28aa9a47e680bee6d5647a3f16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:47:06 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:44736
I20260928 04:47:06.554456 20261 tablet_service.cc:1514] Processing DeleteTablet for tablet d2c80a28aa9a47e680bee6d5647a3f16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:47:06 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:44580
I20260928 04:47:06.554687 20395 tablet_service.cc:1514] Processing DeleteTablet for tablet d2c80a28aa9a47e680bee6d5647a3f16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:47:06 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:34480
I20260928 04:47:06.554913 20503 tablet_replica.cc:330] T d2c80a28aa9a47e680bee6d5647a3f16 P f5624cd530ae4f55add98837eb0efad7: stopping tablet replica
I20260928 04:47:06.555025 20503 raft_consensus.cc:2229] T d2c80a28aa9a47e680bee6d5647a3f16 P f5624cd530ae4f55add98837eb0efad7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:47:06.555145 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 20064
I20260928 04:47:06.555969 20503 raft_consensus.cc:2258] T d2c80a28aa9a47e680bee6d5647a3f16 P f5624cd530ae4f55add98837eb0efad7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:47:06.556740 20503 ts_tablet_manager.cc:1905] T d2c80a28aa9a47e680bee6d5647a3f16 P f5624cd530ae4f55add98837eb0efad7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:47:06.557049 20501 tablet_replica.cc:330] T d2c80a28aa9a47e680bee6d5647a3f16 P b94893b7379e4b098a7161653a42857e: stopping tablet replica
I20260928 04:47:06.557180 20501 raft_consensus.cc:2229] T d2c80a28aa9a47e680bee6d5647a3f16 P b94893b7379e4b098a7161653a42857e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:47:06.557309 20501 raft_consensus.cc:2258] T d2c80a28aa9a47e680bee6d5647a3f16 P b94893b7379e4b098a7161653a42857e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:47:06.557744 20501 ts_tablet_manager.cc:1905] T d2c80a28aa9a47e680bee6d5647a3f16 P b94893b7379e4b098a7161653a42857e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:47:06.558928 20503 ts_tablet_manager.cc:1918] T d2c80a28aa9a47e680bee6d5647a3f16 P f5624cd530ae4f55add98837eb0efad7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:47:06.559005 20503 log.cc:1196] T d2c80a28aa9a47e680bee6d5647a3f16 P f5624cd530ae4f55add98837eb0efad7: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-2/wal/wals/d2c80a28aa9a47e680bee6d5647a3f16
I20260928 04:47:06.559232 20501 ts_tablet_manager.cc:1918] T d2c80a28aa9a47e680bee6d5647a3f16 P b94893b7379e4b098a7161653a42857e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:47:06.559278 20503 ts_tablet_manager.cc:1939] T d2c80a28aa9a47e680bee6d5647a3f16 P f5624cd530ae4f55add98837eb0efad7: Deleting consensus metadata
I20260928 04:47:06.559324 20501 log.cc:1196] T d2c80a28aa9a47e680bee6d5647a3f16 P b94893b7379e4b098a7161653a42857e: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570585693787-1931-0/minicluster-data/ts-1/wal/wals/d2c80a28aa9a47e680bee6d5647a3f16
I20260928 04:47:06.559566 20501 ts_tablet_manager.cc:1939] T d2c80a28aa9a47e680bee6d5647a3f16 P b94893b7379e4b098a7161653a42857e: Deleting consensus metadata
I20260928 04:47:06.559731 19989 catalog_manager.cc:4841] TS f5624cd530ae4f55add98837eb0efad7 (127.1.226.195:36203): tablet d2c80a28aa9a47e680bee6d5647a3f16 (table test-table [id=15086e8634724b3c9c9c6ce2e9b2c760]) successfully deleted
I20260928 04:47:06.559976 19989 catalog_manager.cc:4841] TS b94893b7379e4b098a7161653a42857e (127.1.226.194:43851): tablet d2c80a28aa9a47e680bee6d5647a3f16 (table test-table [id=15086e8634724b3c9c9c6ce2e9b2c760]) successfully deleted
W20260928 04:47:06.561546 19988 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.226.193:38689 (error 108)
I20260928 04:47:06.561782 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 20198
I20260928 04:47:06.562362 20473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:06.561930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:38689 (local address 127.0.0.1:44760)
0928 04:47:06.562283 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:06.562325 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.193:38689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330}
W20260928 04:47:06.562654 19988 catalog_manager.cc:4584] TS 63e2b1f1a9704315a28eadbfa2469c68 (127.1.226.193:38689): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d2c80a28aa9a47e680bee6d5647a3f16: Network error: Client connection negotiation failed: client connection to 127.1.226.193:38689: connect: Connection refused (error 111)
I20260928 04:47:06.566870 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 20332
I20260928 04:47:06.571933 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 19970
2026-09-28T04:47:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1989 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T04:47:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:47:06Z Disabled control of system clock
I20260928 04:47:06.599165 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:39613
--webserver_interface=127.1.226.254
--webserver_port=0
--builtin_ntp_servers=127.1.226.212:43471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:39613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 04:47:06.679170 20508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:06.679389 20508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:06.679410 20508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:06.680513 20508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:06.680547 20508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:06.680560 20508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:06.680572 20508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:06.680584 20508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:06.681840 20508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:43471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:39613
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:39613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:06.682030 20508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:06.682255 20508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:06.687134 20514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:06.687247 20508 server_base.cc:1034] running on GCE node
W20260928 04:47:06.687179 20513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:06.687148 20516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:06.687696 20508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:06.688006 20508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:06.689160 20508 hybrid_clock.cc:648] HybridClock initialized: now 1790570826689132 us; error 49 us; skew 500 ppm
I20260928 04:47:06.690475 20508 webserver.cc:460] Webserver started at http://127.1.226.254:37617/ using document root <none> and password file <none>
I20260928 04:47:06.690763 20508 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:06.690847 20508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:06.691009 20508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:06.692232 20508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "26b164f56dfd43cca5e4e70675273641"
format_stamp: "Formatted at 2026-09-28 04:47:06 on dist-test-slave-j5c3"
server_key: "d501e2129c56701b0368d5a046c475a7"
server_key_iv: "79c686d818f9cf6e89a1f5885ba4c283"
server_key_version: "encryptionkey@0"
I20260928 04:47:06.692581 20508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "26b164f56dfd43cca5e4e70675273641"
format_stamp: "Formatted at 2026-09-28 04:47:06 on dist-test-slave-j5c3"
server_key: "d501e2129c56701b0368d5a046c475a7"
server_key_iv: "79c686d818f9cf6e89a1f5885ba4c283"
server_key_version: "encryptionkey@0"
I20260928 04:47:06.694034 20508 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:47:06.694717 20522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:06.694882 20508 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:47:06.694954 20508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "26b164f56dfd43cca5e4e70675273641"
format_stamp: "Formatted at 2026-09-28 04:47:06 on dist-test-slave-j5c3"
server_key: "d501e2129c56701b0368d5a046c475a7"
server_key_iv: "79c686d818f9cf6e89a1f5885ba4c283"
server_key_version: "encryptionkey@0"
I20260928 04:47:06.695010 20508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:06.708693 20508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:06.709008 20508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:06.709126 20508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:06.713632 20508 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:39613
I20260928 04:47:06.713688 20574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:39613 every 8 connection(s)
I20260928 04:47:06.714082 20508 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:47:06.715127 20575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:06.717576 20575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26b164f56dfd43cca5e4e70675273641: Bootstrap starting.
I20260928 04:47:06.718159 20575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26b164f56dfd43cca5e4e70675273641: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:06.718469 20575 log.cc:824] T 00000000000000000000000000000000 P 26b164f56dfd43cca5e4e70675273641: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:06.719228 20575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26b164f56dfd43cca5e4e70675273641: No bootstrap required, opened a new log
I20260928 04:47:06.720909 20575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26b164f56dfd43cca5e4e70675273641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26b164f56dfd43cca5e4e70675273641" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 39613 } }
I20260928 04:47:06.721071 20575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26b164f56dfd43cca5e4e70675273641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:06.721118 20575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26b164f56dfd43cca5e4e70675273641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b164f56dfd43cca5e4e70675273641, State: Initialized, Role: FOLLOWER
I20260928 04:47:06.721287 20575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26b164f56dfd43cca5e4e70675273641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26b164f56dfd43cca5e4e70675273641" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 39613 } }
I20260928 04:47:06.721395 20575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26b164f56dfd43cca5e4e70675273641 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:47:06.721439 20575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26b164f56dfd43cca5e4e70675273641 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:47:06.721488 20575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26b164f56dfd43cca5e4e70675273641 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:06.721990 20575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26b164f56dfd43cca5e4e70675273641 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26b164f56dfd43cca5e4e70675273641" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 39613 } }
I20260928 04:47:06.722132 20575 leader_election.cc:302] T 00000000000000000000000000000000 P 26b164f56dfd43cca5e4e70675273641 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26b164f56dfd43cca5e4e70675273641; no voters:
I20260928 04:47:06.722340 20575 leader_election.cc:288] T 00000000000000000000000000000000 P 26b164f56dfd43cca5e4e70675273641 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:47:06.722402 20578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26b164f56dfd43cca5e4e70675273641 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:06.722609 20578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26b164f56dfd43cca5e4e70675273641 [term 1 LEADER]: Becoming Leader. State: Replica: 26b164f56dfd43cca5e4e70675273641, State: Running, Role: LEADER
I20260928 04:47:06.722608 20575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26b164f56dfd43cca5e4e70675273641 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:06.722785 20578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26b164f56dfd43cca5e4e70675273641 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26b164f56dfd43cca5e4e70675273641" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 39613 } }
I20260928 04:47:06.723198 20580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26b164f56dfd43cca5e4e70675273641 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26b164f56dfd43cca5e4e70675273641. Latest consensus state: current_term: 1 leader_uuid: "26b164f56dfd43cca5e4e70675273641" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26b164f56dfd43cca5e4e70675273641" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 39613 } } }
I20260928 04:47:06.723276 20580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26b164f56dfd43cca5e4e70675273641 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:06.723300 20579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26b164f56dfd43cca5e4e70675273641 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26b164f56dfd43cca5e4e70675273641" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26b164f56dfd43cca5e4e70675273641" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 39613 } } }
I20260928 04:47:06.723390 20579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26b164f56dfd43cca5e4e70675273641 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:06.723492 20586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:47:06.723632 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 20508
I20260928 04:47:06.723738 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:47:06.723999 1931 external_mini_cluster.cc:1428] Setting key ff2bc838b67c5a312942ff8a6cee5f8d
I20260928 04:47:06.724108 20586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:47:06.725881 20586 catalog_manager.cc:1347] Generated new cluster ID: 2aa76eaf02d44e8e9faf57997a9005b7
I20260928 04:47:06.725945 20586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:47:06.730254 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:06.724582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49258 (local address 127.1.226.254:39613)
0928 04:47:06.724833 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:06.724842 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:06.725018 (+ 176us) server_negotiation.cc:407] Connection header received
0928 04:47:06.725116 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:06.725121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:06.725184 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:06.725332 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:06.725538 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.726912 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.728042 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.728696 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.728804 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:06.729555 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:06.729559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:06.729559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:06.729759 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:06.729797 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:06.729810 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:06.729877 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:06.729982 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 04:47:06.730058 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":87,"threads_started":1}
I20260928 04:47:06.754822 20586 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:47:06.755360 20586 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:47:06.761725 20586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26b164f56dfd43cca5e4e70675273641: Generated new TSK 0
I20260928 04:47:06.762017 20586 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:47:06.769402 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:0
--local_ip_for_outbound_sockets=127.1.226.193
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:39613
--builtin_ntp_servers=127.1.226.212:43471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 04:47:06.852458 20599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:06.852674 20599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:06.852695 20599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:06.853879 20599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:06.853928 20599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:06.853955 20599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:47:06.855352 20599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:43471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:39613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:06.855624 20599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:06.855948 20599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:06.861189 20604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:06.861200 20605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:06.861230 20599 server_base.cc:1034] running on GCE node
W20260928 04:47:06.861195 20607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:06.861621 20599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:06.861900 20599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:06.863075 20599 hybrid_clock.cc:648] HybridClock initialized: now 1790570826863059 us; error 28 us; skew 500 ppm
I20260928 04:47:06.868921 20599 webserver.cc:460] Webserver started at http://127.1.226.193:38669/ using document root <none> and password file <none>
I20260928 04:47:06.869231 20599 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:06.869318 20599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:06.869501 20599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:06.870649 20599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-0/data/instance:
uuid: "d55c397758ce4f91a83c2f59446b0392"
format_stamp: "Formatted at 2026-09-28 04:47:06 on dist-test-slave-j5c3"
server_key: "35348cafdd259ed51ce631591941d287"
server_key_iv: "1912e7aa852e557fe526d45c955e689f"
server_key_version: "encryptionkey@0"
I20260928 04:47:06.871032 20599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance:
uuid: "d55c397758ce4f91a83c2f59446b0392"
format_stamp: "Formatted at 2026-09-28 04:47:06 on dist-test-slave-j5c3"
server_key: "35348cafdd259ed51ce631591941d287"
server_key_iv: "1912e7aa852e557fe526d45c955e689f"
server_key_version: "encryptionkey@0"
I20260928 04:47:06.872524 20599 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:47:06.873381 20613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:06.873588 20599 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:47:06.873757 20599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "d55c397758ce4f91a83c2f59446b0392"
format_stamp: "Formatted at 2026-09-28 04:47:06 on dist-test-slave-j5c3"
server_key: "35348cafdd259ed51ce631591941d287"
server_key_iv: "1912e7aa852e557fe526d45c955e689f"
server_key_version: "encryptionkey@0"
I20260928 04:47:06.873845 20599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:06.886507 20599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:06.886900 20599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:06.887064 20599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:06.887305 20599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:47:06.887641 20599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:47:06.887698 20599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:06.887742 20599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:47:06.887764 20599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:06.894094 20599 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:46151
I20260928 04:47:06.894150 20726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:46151 every 8 connection(s)
I20260928 04:47:06.894845 20599 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
I20260928 04:47:06.901059 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:06.895731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:40515 (local address 127.1.226.254:39613)
0928 04:47:06.895843 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:06.895845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:06.896096 (+ 251us) server_negotiation.cc:407] Connection header received
0928 04:47:06.896779 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:06.896781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:06.896798 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:06.896870 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:06.897539 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.898168 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.899306 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.899527 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.899587 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:06.900704 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:06.900705 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:06.900706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:06.900710 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:06.900726 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:06.900731 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:06.900837 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:06.900956 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 04:47:06.900988 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 04:47:06.901252 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:06.895733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:39613 (local address 127.1.226.193:40515)
0928 04:47:06.895959 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:06.895995 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:06.896671 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:06.896928 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:06.896931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:06.896940 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:06.897453 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:06.897457 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.898271 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.898272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:06.899200 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:06.899202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:06.899609 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:06.899610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:06.899727 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:06.900555 (+ 828us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:47:06.900555 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:06.900586 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:47:06.900610 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:06.900852 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:06.900857 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:06.900860 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:47:06.900924 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 04:47:06.901010 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":126,"threads_started":1}
I20260928 04:47:06.901882 20727 heartbeater.cc:347] Connected to a master server at 127.1.226.254:39613
I20260928 04:47:06.901984 20727 heartbeater.cc:464] Registering TS with master...
I20260928 04:47:06.902143 20727 heartbeater.cc:511] Master 127.1.226.254:39613 requested a full tablet report, sending...
I20260928 04:47:06.902792 20539 ts_manager.cc:194] Registered new tserver with Master: d55c397758ce4f91a83c2f59446b0392 (127.1.226.193:46151)
I20260928 04:47:06.903532 20539 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.226.193:40515
I20260928 04:47:06.904675 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 20599
I20260928 04:47:06.904808 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance
I20260928 04:47:06.905035 1931 external_mini_cluster.cc:1428] Setting key 1f1ea685f70fb4ff36cc1b73336bf8ad
I20260928 04:47:06.905954 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:0
--local_ip_for_outbound_sockets=127.1.226.194
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:39613
--builtin_ntp_servers=127.1.226.212:43471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 04:47:06.989332 20730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:06.989563 20730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:06.989585 20730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:06.990839 20730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:06.990881 20730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:06.990907 20730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:47:06.992290 20730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:43471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:39613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:06.992556 20730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:06.992864 20730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:06.998175 20735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:06.998142 20736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:06.998139 20738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:06.998188 20730 server_base.cc:1034] running on GCE node
I20260928 04:47:06.998625 20730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:06.998864 20730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:07.000043 20730 hybrid_clock.cc:648] HybridClock initialized: now 1790570827000033 us; error 30 us; skew 500 ppm
I20260928 04:47:07.001389 20730 webserver.cc:460] Webserver started at http://127.1.226.194:42379/ using document root <none> and password file <none>
I20260928 04:47:07.001675 20730 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:07.001727 20730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:07.001822 20730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:07.002971 20730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-1/data/instance:
uuid: "696c4d9cc2584f089c6c0805f97b8a17"
format_stamp: "Formatted at 2026-09-28 04:47:06 on dist-test-slave-j5c3"
server_key: "f15e922cd5ac5d29f55d7ea09ee6115c"
server_key_iv: "32780069e09343831467c0d5f30f5940"
server_key_version: "encryptionkey@0"
I20260928 04:47:07.003345 20730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance:
uuid: "696c4d9cc2584f089c6c0805f97b8a17"
format_stamp: "Formatted at 2026-09-28 04:47:06 on dist-test-slave-j5c3"
server_key: "f15e922cd5ac5d29f55d7ea09ee6115c"
server_key_iv: "32780069e09343831467c0d5f30f5940"
server_key_version: "encryptionkey@0"
I20260928 04:47:07.004878 20730 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 04:47:07.005721 20744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:07.005927 20730 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:07.006026 20730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "696c4d9cc2584f089c6c0805f97b8a17"
format_stamp: "Formatted at 2026-09-28 04:47:06 on dist-test-slave-j5c3"
server_key: "f15e922cd5ac5d29f55d7ea09ee6115c"
server_key_iv: "32780069e09343831467c0d5f30f5940"
server_key_version: "encryptionkey@0"
I20260928 04:47:07.006088 20730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:07.018790 20730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:07.019088 20730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:07.019203 20730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:07.019407 20730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:47:07.019776 20730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:47:07.019815 20730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:07.019840 20730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:47:07.019881 20730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:07.026604 20730 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:40319
I20260928 04:47:07.026680 20857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:40319 every 8 connection(s)
I20260928 04:47:07.027070 20730 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
I20260928 04:47:07.030927 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 20730
I20260928 04:47:07.031018 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance
I20260928 04:47:07.031178 1931 external_mini_cluster.cc:1428] Setting key db74b806ff867703df77548ab4cc3b76
I20260928 04:47:07.032253 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:0
--local_ip_for_outbound_sockets=127.1.226.195
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:39613
--builtin_ntp_servers=127.1.226.212:43471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 04:47:07.033113 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:07.028074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:39025 (local address 127.1.226.254:39613)
0928 04:47:07.028178 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:07.028180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:07.028357 (+ 177us) server_negotiation.cc:407] Connection header received
0928 04:47:07.028959 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:07.028960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:07.028973 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:07.029032 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:07.029787 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.030293 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.031472 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.031768 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.031856 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:07.032837 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:07.032838 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:07.032839 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:07.032843 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:07.032859 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:07.032863 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:07.032915 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:07.033004 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 04:47:07.033034 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 04:47:07.033442 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:07.027849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:39613 (local address 127.1.226.194:39025)
0928 04:47:07.028078 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:07.028153 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:07.028832 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:07.029109 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:07.029114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:07.029124 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:07.029680 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:07.029685 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.030394 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.030395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:07.031373 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:07.031376 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.031863 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.031864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:07.032022 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:07.032659 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:47:07.032659 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:07.032689 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:47:07.032714 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:07.032924 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:07.032930 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:07.032933 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:47:07.032998 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 04:47:07.033104 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":69,"threads_started":1}
I20260928 04:47:07.033968 20858 heartbeater.cc:347] Connected to a master server at 127.1.226.254:39613
I20260928 04:47:07.034085 20858 heartbeater.cc:464] Registering TS with master...
I20260928 04:47:07.034269 20858 heartbeater.cc:511] Master 127.1.226.254:39613 requested a full tablet report, sending...
I20260928 04:47:07.034678 20539 ts_manager.cc:194] Registered new tserver with Master: 696c4d9cc2584f089c6c0805f97b8a17 (127.1.226.194:40319)
I20260928 04:47:07.035164 20539 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.226.194:39025
W20260928 04:47:07.115792 20861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:07.116014 20861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:07.116035 20861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:07.117238 20861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:07.117283 20861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:07.117308 20861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:47:07.118674 20861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:43471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:39613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:07.118953 20861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:07.119277 20861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:07.124833 20866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:07.124917 20869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:07.124852 20867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:07.125347 20861 server_base.cc:1034] running on GCE node
I20260928 04:47:07.125574 20861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:07.125808 20861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:07.126972 20861 hybrid_clock.cc:648] HybridClock initialized: now 1790570827126950 us; error 40 us; skew 500 ppm
I20260928 04:47:07.128450 20861 webserver.cc:460] Webserver started at http://127.1.226.195:36513/ using document root <none> and password file <none>
I20260928 04:47:07.128722 20861 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:07.128775 20861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:07.128957 20861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:07.130132 20861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-2/data/instance:
uuid: "35e24b8388fe4d83be91a8843f0fc883"
format_stamp: "Formatted at 2026-09-28 04:47:07 on dist-test-slave-j5c3"
server_key: "0fbb551005229a53fa46106459d109ac"
server_key_iv: "2ca4b5dac96a1516d3bc3eb41de77c43"
server_key_version: "encryptionkey@0"
I20260928 04:47:07.130534 20861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance:
uuid: "35e24b8388fe4d83be91a8843f0fc883"
format_stamp: "Formatted at 2026-09-28 04:47:07 on dist-test-slave-j5c3"
server_key: "0fbb551005229a53fa46106459d109ac"
server_key_iv: "2ca4b5dac96a1516d3bc3eb41de77c43"
server_key_version: "encryptionkey@0"
I20260928 04:47:07.132018 20861 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:47:07.132881 20875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:07.133064 20861 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:47:07.133183 20861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "35e24b8388fe4d83be91a8843f0fc883"
format_stamp: "Formatted at 2026-09-28 04:47:07 on dist-test-slave-j5c3"
server_key: "0fbb551005229a53fa46106459d109ac"
server_key_iv: "2ca4b5dac96a1516d3bc3eb41de77c43"
server_key_version: "encryptionkey@0"
I20260928 04:47:07.133271 20861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:07.144769 20861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:07.145123 20861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:07.145298 20861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:07.145561 20861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:47:07.145927 20861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:47:07.145992 20861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:07.146034 20861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:47:07.146066 20861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:07.152642 20861 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:37241
I20260928 04:47:07.152688 20988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:37241 every 8 connection(s)
I20260928 04:47:07.153079 20861 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
I20260928 04:47:07.157745 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 20861
I20260928 04:47:07.157851 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance
I20260928 04:47:07.158320 1931 external_mini_cluster.cc:1428] Setting key 25917f3a2f08b079d06c3a4e73fb2386
I20260928 04:47:07.158727 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:07.153948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:44587 (local address 127.1.226.254:39613)
0928 04:47:07.153998 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:07.154000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:07.154253 (+ 253us) server_negotiation.cc:407] Connection header received
0928 04:47:07.154777 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:07.154778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:07.154790 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:07.154863 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:07.155609 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.156137 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.157272 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.157462 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.157538 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:07.158399 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:07.158401 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:07.158401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:07.158407 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:07.158427 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:07.158431 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:07.158488 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:07.158639 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 04:47:07.158668 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 04:47:07.159024 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:07.153892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:39613 (local address 127.1.226.195:44587)
0928 04:47:07.154099 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:07.154147 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:07.154677 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:07.154946 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:07.154950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:07.154958 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:07.155519 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:07.155524 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.156274 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.156275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:07.157174 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:07.157177 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.157542 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.157543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:07.157663 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:07.158243 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:47:07.158243 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:07.158275 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:47:07.158299 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:07.158512 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:07.158518 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:07.158521 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:47:07.158629 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 04:47:07.158760 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":106,"threads_started":1}
I20260928 04:47:07.159544 20989 heartbeater.cc:347] Connected to a master server at 127.1.226.254:39613
I20260928 04:47:07.159626 20989 heartbeater.cc:464] Registering TS with master...
I20260928 04:47:07.159770 20989 heartbeater.cc:511] Master 127.1.226.254:39613 requested a full tablet report, sending...
I20260928 04:47:07.160085 20539 ts_manager.cc:194] Registered new tserver with Master: 35e24b8388fe4d83be91a8843f0fc883 (127.1.226.195:37241)
I20260928 04:47:07.160527 20539 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.226.195:44587
I20260928 04:47:07.170653 1931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:47:07.175282 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:07.172106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49274 (local address 127.1.226.254:39613)
0928 04:47:07.172210 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:07.172214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:07.172347 (+ 133us) server_negotiation.cc:407] Connection header received
0928 04:47:07.172436 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:07.172437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:07.172448 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:07.172508 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:07.172680 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.173197 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.173967 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.174161 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.174234 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:07.174980 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:07.174982 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:07.174982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:07.174987 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:07.175007 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:07.175010 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:07.175109 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:07.175181 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 04:47:07.175192 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 04:47:07.176978 20539 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:47:07.178592 20539 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:47:07.185544 20998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:07.180969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:46151 (local address 127.0.0.1:50754)
0928 04:47:07.181275 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:07.181323 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:07.181430 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:07.182101 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:07.182101 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:07.182103 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:07.182142 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:07.182144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.183023 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.183025 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:07.184237 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:07.184239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.185240 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.185241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:07.185319 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:07.185334 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 04:47:07.185404 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 04:47:07.185455 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":94,"threads_started":1}
I20260928 04:47:07.185568 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:07.181126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50754 (local address 127.1.226.193:46151)
0928 04:47:07.181678 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:07.181682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:07.181691 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:47:07.181804 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:07.181806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:07.181820 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:07.181898 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:07.182225 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.182929 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.184326 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.185154 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.185247 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:07.185355 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:07.185420 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 04:47:07.185477 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":97,"threads_started":1}
I20260928 04:47:07.185925 21001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:07.181251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59160 (local address 127.1.226.194:40319)
0928 04:47:07.181451 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:07.181457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:07.181520 (+ 63us) server_negotiation.cc:407] Connection header received
0928 04:47:07.181604 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:07.181607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:07.181626 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:07.181716 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:07.182087 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.183347 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.184634 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.185419 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.185489 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:07.185581 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:07.185796 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 04:47:07.185838 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":94,"threads_started":1}
I20260928 04:47:07.185900 21000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:07.180969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:40319 (local address 127.0.0.1:59160)
0928 04:47:07.181434 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:07.181445 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:07.181526 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:07.181936 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:07.181937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:07.181940 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:07.181990 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:07.181992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.183443 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.183444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:07.184546 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:07.184547 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.185636 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.185637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:07.185712 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:07.185716 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:47:07.185777 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 04:47:07.185822 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":4,"thread_start_us":93,"threads_started":1}
I20260928 04:47:07.186386 20997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:07.181210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:37241 (local address 127.0.0.1:33640)
0928 04:47:07.181553 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:07.181563 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:07.181604 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:07.181879 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:07.181883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:07.181885 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:07.181957 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:07.181959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.184344 (+ 2385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.184345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:07.185106 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:07.185108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.186150 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.186151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:07.186226 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:07.186230 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:47:07.186281 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 04:47:07.186323 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":76,"threads_started":1}
I20260928 04:47:07.186666 21002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:07.181435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33640 (local address 127.1.226.195:37241)
0928 04:47:07.181622 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:07.181628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:07.181649 (+ 21us) server_negotiation.cc:407] Connection header received
0928 04:47:07.181692 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:07.181694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:07.181710 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:07.181795 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:07.182033 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.183352 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.185209 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.185956 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.186022 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:07.186113 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:07.186543 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 04:47:07.186589 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":80,"threads_started":1}
I20260928 04:47:07.186875 20661 tablet_service.cc:1467] Processing CreateTablet for tablet b773659dc0cd4d2b9c9c573e8d831e44 (DEFAULT_TABLE table=test-table [id=b8faea945a4447d2ae6c6ff25654d8f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:47:07.186925 20792 tablet_service.cc:1467] Processing CreateTablet for tablet b773659dc0cd4d2b9c9c573e8d831e44 (DEFAULT_TABLE table=test-table [id=b8faea945a4447d2ae6c6ff25654d8f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:47:07.187244 20792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b773659dc0cd4d2b9c9c573e8d831e44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:07.187248 20661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b773659dc0cd4d2b9c9c573e8d831e44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:07.187345 20923 tablet_service.cc:1467] Processing CreateTablet for tablet b773659dc0cd4d2b9c9c573e8d831e44 (DEFAULT_TABLE table=test-table [id=b8faea945a4447d2ae6c6ff25654d8f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:47:07.187625 20923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b773659dc0cd4d2b9c9c573e8d831e44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:07.190284 21003 tablet_bootstrap.cc:492] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17: Bootstrap starting.
I20260928 04:47:07.190774 21005 tablet_bootstrap.cc:492] T b773659dc0cd4d2b9c9c573e8d831e44 P 35e24b8388fe4d83be91a8843f0fc883: Bootstrap starting.
I20260928 04:47:07.191035 21003 tablet_bootstrap.cc:654] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:07.191195 21004 tablet_bootstrap.cc:492] T b773659dc0cd4d2b9c9c573e8d831e44 P d55c397758ce4f91a83c2f59446b0392: Bootstrap starting.
I20260928 04:47:07.191412 21003 log.cc:824] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:07.191453 21005 tablet_bootstrap.cc:654] T b773659dc0cd4d2b9c9c573e8d831e44 P 35e24b8388fe4d83be91a8843f0fc883: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:07.191787 21005 log.cc:824] T b773659dc0cd4d2b9c9c573e8d831e44 P 35e24b8388fe4d83be91a8843f0fc883: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:07.192072 21004 tablet_bootstrap.cc:654] T b773659dc0cd4d2b9c9c573e8d831e44 P d55c397758ce4f91a83c2f59446b0392: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:07.192271 21003 tablet_bootstrap.cc:492] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17: No bootstrap required, opened a new log
I20260928 04:47:07.192348 21003 ts_tablet_manager.cc:1397] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:47:07.192431 21004 log.cc:824] T b773659dc0cd4d2b9c9c573e8d831e44 P d55c397758ce4f91a83c2f59446b0392: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:07.192627 21005 tablet_bootstrap.cc:492] T b773659dc0cd4d2b9c9c573e8d831e44 P 35e24b8388fe4d83be91a8843f0fc883: No bootstrap required, opened a new log
I20260928 04:47:07.192700 21005 ts_tablet_manager.cc:1397] T b773659dc0cd4d2b9c9c573e8d831e44 P 35e24b8388fe4d83be91a8843f0fc883: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:47:07.193306 21004 tablet_bootstrap.cc:492] T b773659dc0cd4d2b9c9c573e8d831e44 P d55c397758ce4f91a83c2f59446b0392: No bootstrap required, opened a new log
I20260928 04:47:07.193391 21004 ts_tablet_manager.cc:1397] T b773659dc0cd4d2b9c9c573e8d831e44 P d55c397758ce4f91a83c2f59446b0392: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 04:47:07.195099 21003 raft_consensus.cc:348] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55c397758ce4f91a83c2f59446b0392" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 46151 } } peers { permanent_uuid: "696c4d9cc2584f089c6c0805f97b8a17" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40319 } } peers { permanent_uuid: "35e24b8388fe4d83be91a8843f0fc883" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 37241 } }
I20260928 04:47:07.195286 21003 raft_consensus.cc:374] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:07.195323 21003 raft_consensus.cc:729] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696c4d9cc2584f089c6c0805f97b8a17, State: Initialized, Role: FOLLOWER
I20260928 04:47:07.195446 21003 consensus_queue.cc:260] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55c397758ce4f91a83c2f59446b0392" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 46151 } } peers { permanent_uuid: "696c4d9cc2584f089c6c0805f97b8a17" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40319 } } peers { permanent_uuid: "35e24b8388fe4d83be91a8843f0fc883" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 37241 } }
I20260928 04:47:07.195353 21005 raft_consensus.cc:348] T b773659dc0cd4d2b9c9c573e8d831e44 P 35e24b8388fe4d83be91a8843f0fc883 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55c397758ce4f91a83c2f59446b0392" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 46151 } } peers { permanent_uuid: "696c4d9cc2584f089c6c0805f97b8a17" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40319 } } peers { permanent_uuid: "35e24b8388fe4d83be91a8843f0fc883" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 37241 } }
I20260928 04:47:07.195600 21005 raft_consensus.cc:374] T b773659dc0cd4d2b9c9c573e8d831e44 P 35e24b8388fe4d83be91a8843f0fc883 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:07.195647 21005 raft_consensus.cc:729] T b773659dc0cd4d2b9c9c573e8d831e44 P 35e24b8388fe4d83be91a8843f0fc883 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35e24b8388fe4d83be91a8843f0fc883, State: Initialized, Role: FOLLOWER
I20260928 04:47:07.195714 21003 ts_tablet_manager.cc:1428] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:47:07.195832 20858 heartbeater.cc:503] Master 127.1.226.254:39613 was elected leader, sending a full tablet report...
I20260928 04:47:07.195788 21005 consensus_queue.cc:260] T b773659dc0cd4d2b9c9c573e8d831e44 P 35e24b8388fe4d83be91a8843f0fc883 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55c397758ce4f91a83c2f59446b0392" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 46151 } } peers { permanent_uuid: "696c4d9cc2584f089c6c0805f97b8a17" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40319 } } peers { permanent_uuid: "35e24b8388fe4d83be91a8843f0fc883" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 37241 } }
I20260928 04:47:07.195899 21004 raft_consensus.cc:348] T b773659dc0cd4d2b9c9c573e8d831e44 P d55c397758ce4f91a83c2f59446b0392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55c397758ce4f91a83c2f59446b0392" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 46151 } } peers { permanent_uuid: "696c4d9cc2584f089c6c0805f97b8a17" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40319 } } peers { permanent_uuid: "35e24b8388fe4d83be91a8843f0fc883" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 37241 } }
I20260928 04:47:07.196044 21004 raft_consensus.cc:374] T b773659dc0cd4d2b9c9c573e8d831e44 P d55c397758ce4f91a83c2f59446b0392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:07.196067 21005 ts_tablet_manager.cc:1428] T b773659dc0cd4d2b9c9c573e8d831e44 P 35e24b8388fe4d83be91a8843f0fc883: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:47:07.196089 21004 raft_consensus.cc:729] T b773659dc0cd4d2b9c9c573e8d831e44 P d55c397758ce4f91a83c2f59446b0392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d55c397758ce4f91a83c2f59446b0392, State: Initialized, Role: FOLLOWER
I20260928 04:47:07.196244 21004 consensus_queue.cc:260] T b773659dc0cd4d2b9c9c573e8d831e44 P d55c397758ce4f91a83c2f59446b0392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55c397758ce4f91a83c2f59446b0392" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 46151 } } peers { permanent_uuid: "696c4d9cc2584f089c6c0805f97b8a17" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40319 } } peers { permanent_uuid: "35e24b8388fe4d83be91a8843f0fc883" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 37241 } }
I20260928 04:47:07.196398 20989 heartbeater.cc:503] Master 127.1.226.254:39613 was elected leader, sending a full tablet report...
I20260928 04:47:07.196497 21004 ts_tablet_manager.cc:1428] T b773659dc0cd4d2b9c9c573e8d831e44 P d55c397758ce4f91a83c2f59446b0392: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 04:47:07.196594 20727 heartbeater.cc:503] Master 127.1.226.254:39613 was elected leader, sending a full tablet report...
W20260928 04:47:07.277628 20859 tablet.cc:2369] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:47:07.356372 21009 raft_consensus.cc:482] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:47:07.356540 21009 raft_consensus.cc:504] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55c397758ce4f91a83c2f59446b0392" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 46151 } } peers { permanent_uuid: "696c4d9cc2584f089c6c0805f97b8a17" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40319 } } peers { permanent_uuid: "35e24b8388fe4d83be91a8843f0fc883" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 37241 } }
I20260928 04:47:07.356953 21009 leader_election.cc:288] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d55c397758ce4f91a83c2f59446b0392 (127.1.226.193:46151), 35e24b8388fe4d83be91a8843f0fc883 (127.1.226.195:37241)
I20260928 04:47:07.359961 21002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:07.357429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:47401 (local address 127.1.226.195:37241)
0928 04:47:07.357498 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:07.357500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:07.357510 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:47:07.357587 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:07.357588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:07.357596 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:07.357660 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:07.357833 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.358296 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.359257 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.359593 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.359684 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:07.359692 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:07.359849 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 04:47:07.359889 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 04:47:07.359970 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:07.357163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:46151 (local address 127.1.226.194:60719)
0928 04:47:07.357227 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:07.357265 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:07.357316 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:07.357528 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:07.357529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:07.357531 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:07.357609 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:07.357612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.358323 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.358324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:07.359108 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:07.359110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.359687 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.359688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:07.359770 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:07.359774 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:47:07.359858 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 04:47:07.359898 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":12}
I20260928 04:47:07.359978 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:07.357264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:60719 (local address 127.1.226.193:46151)
0928 04:47:07.357378 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:07.357380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:07.357390 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:47:07.357423 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:07.357424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:07.357436 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:07.357520 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:07.357695 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.358236 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.359207 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.359596 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.359688 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:07.359696 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:07.359878 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 04:47:07.359916 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 04:47:07.359985 21012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:07.357296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:37241 (local address 127.1.226.194:47401)
0928 04:47:07.357458 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:07.357468 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:07.357516 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:07.357692 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:07.357693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:07.357694 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:07.357756 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:07.357758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.358383 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.358384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:07.359131 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:07.359133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.359697 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.359698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:07.359770 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:07.359774 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:47:07.359822 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 04:47:07.359889 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":102,"threads_started":1}
I20260928 04:47:07.360559 20943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b773659dc0cd4d2b9c9c573e8d831e44" candidate_uuid: "696c4d9cc2584f089c6c0805f97b8a17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35e24b8388fe4d83be91a8843f0fc883" is_pre_election: true
I20260928 04:47:07.360586 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b773659dc0cd4d2b9c9c573e8d831e44" candidate_uuid: "696c4d9cc2584f089c6c0805f97b8a17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d55c397758ce4f91a83c2f59446b0392" is_pre_election: true
I20260928 04:47:07.360688 20943 raft_consensus.cc:2454] T b773659dc0cd4d2b9c9c573e8d831e44 P 35e24b8388fe4d83be91a8843f0fc883 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696c4d9cc2584f089c6c0805f97b8a17 in term 0.
I20260928 04:47:07.360746 20681 raft_consensus.cc:2454] T b773659dc0cd4d2b9c9c573e8d831e44 P d55c397758ce4f91a83c2f59446b0392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696c4d9cc2584f089c6c0805f97b8a17 in term 0.
I20260928 04:47:07.360953 20747 leader_election.cc:302] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35e24b8388fe4d83be91a8843f0fc883, 696c4d9cc2584f089c6c0805f97b8a17; no voters:
I20260928 04:47:07.361117 21009 raft_consensus.cc:2781] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:47:07.361202 21009 raft_consensus.cc:482] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:47:07.361219 21009 raft_consensus.cc:3037] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:07.361995 21009 raft_consensus.cc:504] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55c397758ce4f91a83c2f59446b0392" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 46151 } } peers { permanent_uuid: "696c4d9cc2584f089c6c0805f97b8a17" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40319 } } peers { permanent_uuid: "35e24b8388fe4d83be91a8843f0fc883" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 37241 } }
I20260928 04:47:07.362102 21009 leader_election.cc:288] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17 [CANDIDATE]: Term 1 election: Requested vote from peers d55c397758ce4f91a83c2f59446b0392 (127.1.226.193:46151), 35e24b8388fe4d83be91a8843f0fc883 (127.1.226.195:37241)
I20260928 04:47:07.362315 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b773659dc0cd4d2b9c9c573e8d831e44" candidate_uuid: "696c4d9cc2584f089c6c0805f97b8a17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d55c397758ce4f91a83c2f59446b0392"
I20260928 04:47:07.362347 20943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b773659dc0cd4d2b9c9c573e8d831e44" candidate_uuid: "696c4d9cc2584f089c6c0805f97b8a17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35e24b8388fe4d83be91a8843f0fc883"
I20260928 04:47:07.362396 20681 raft_consensus.cc:3037] T b773659dc0cd4d2b9c9c573e8d831e44 P d55c397758ce4f91a83c2f59446b0392 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:07.362430 20943 raft_consensus.cc:3037] T b773659dc0cd4d2b9c9c573e8d831e44 P 35e24b8388fe4d83be91a8843f0fc883 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:07.363222 20681 raft_consensus.cc:2454] T b773659dc0cd4d2b9c9c573e8d831e44 P d55c397758ce4f91a83c2f59446b0392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696c4d9cc2584f089c6c0805f97b8a17 in term 1.
I20260928 04:47:07.363389 20747 leader_election.cc:302] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696c4d9cc2584f089c6c0805f97b8a17, d55c397758ce4f91a83c2f59446b0392; no voters:
I20260928 04:47:07.363440 20943 raft_consensus.cc:2454] T b773659dc0cd4d2b9c9c573e8d831e44 P 35e24b8388fe4d83be91a8843f0fc883 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696c4d9cc2584f089c6c0805f97b8a17 in term 1.
I20260928 04:47:07.363500 21009 raft_consensus.cc:2781] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:07.363659 21009 raft_consensus.cc:686] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17 [term 1 LEADER]: Becoming Leader. State: Replica: 696c4d9cc2584f089c6c0805f97b8a17, State: Running, Role: LEADER
I20260928 04:47:07.363765 21009 consensus_queue.cc:237] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55c397758ce4f91a83c2f59446b0392" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 46151 } } peers { permanent_uuid: "696c4d9cc2584f089c6c0805f97b8a17" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40319 } } peers { permanent_uuid: "35e24b8388fe4d83be91a8843f0fc883" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 37241 } }
I20260928 04:47:07.364552 20539 catalog_manager.cc:5495] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696c4d9cc2584f089c6c0805f97b8a17 (127.1.226.194). New cstate: current_term: 1 leader_uuid: "696c4d9cc2584f089c6c0805f97b8a17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55c397758ce4f91a83c2f59446b0392" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 46151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696c4d9cc2584f089c6c0805f97b8a17" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 40319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35e24b8388fe4d83be91a8843f0fc883" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 37241 } health_report { overall_health: UNKNOWN } } }
W20260928 04:47:07.395689 20728 tablet.cc:2369] T b773659dc0cd4d2b9c9c573e8d831e44 P d55c397758ce4f91a83c2f59446b0392: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:47:07.403708 20990 tablet.cc:2369] T b773659dc0cd4d2b9c9c573e8d831e44 P 35e24b8388fe4d83be91a8843f0fc883: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:47:07.412768 21001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:07.410351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59168 (local address 127.1.226.194:40319)
0928 04:47:07.410446 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:07.410447 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:07.410454 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:47:07.410487 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:07.410488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:07.410507 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:07.410587 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:47:07.410713 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.411177 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.411943 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:07.412137 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:07.412262 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:07.412415 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:47:07.412509 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:47:07.412620 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:07.412650 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 04:47:07.412659 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 04:47:07.414804 20681 raft_consensus.cc:1261] T b773659dc0cd4d2b9c9c573e8d831e44 P d55c397758ce4f91a83c2f59446b0392 [term 1 FOLLOWER]: Refusing update from remote peer 696c4d9cc2584f089c6c0805f97b8a17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:47:07.414861 20943 raft_consensus.cc:1261] T b773659dc0cd4d2b9c9c573e8d831e44 P 35e24b8388fe4d83be91a8843f0fc883 [term 1 FOLLOWER]: Refusing update from remote peer 696c4d9cc2584f089c6c0805f97b8a17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:47:07.415256 21013 consensus_queue.cc:1035] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d55c397758ce4f91a83c2f59446b0392" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:47:07.415396 21009 consensus_queue.cc:1035] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35e24b8388fe4d83be91a8843f0fc883" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 37241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:47:07.417910 21017 mvcc.cc:205] Tried to move back new op lower bound from 7334178109087940608 to 7334178108882481152. Current Snapshot: MvccSnapshot[applied={T|T < 7334178109087940608}]
I20260928 04:47:07.419360 21018 mvcc.cc:205] Tried to move back new op lower bound from 7334178109087940608 to 7334178108882481152. Current Snapshot: MvccSnapshot[applied={T|T < 7334178109087940608 or (T in {7334178109087940608})}]
I20260928 04:47:07.420192 20538 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:47:07.420306 20538 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:47:07.421277 20538 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26b164f56dfd43cca5e4e70675273641: Sending DeleteTablet for 3 replicas of tablet b773659dc0cd4d2b9c9c573e8d831e44
I20260928 04:47:07.421599 20792 tablet_service.cc:1514] Processing DeleteTablet for tablet b773659dc0cd4d2b9c9c573e8d831e44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:47:07 UTC) from {username='slave'} at 127.0.0.1:59160
I20260928 04:47:07.421653 20923 tablet_service.cc:1514] Processing DeleteTablet for tablet b773659dc0cd4d2b9c9c573e8d831e44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:47:07 UTC) from {username='slave'} at 127.0.0.1:33640
I20260928 04:47:07.421890 21026 tablet_replica.cc:330] T b773659dc0cd4d2b9c9c573e8d831e44 P 35e24b8388fe4d83be91a8843f0fc883: stopping tablet replica
I20260928 04:47:07.421998 21026 raft_consensus.cc:2229] T b773659dc0cd4d2b9c9c573e8d831e44 P 35e24b8388fe4d83be91a8843f0fc883 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:47:07.422000 20661 tablet_service.cc:1514] Processing DeleteTablet for tablet b773659dc0cd4d2b9c9c573e8d831e44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:47:07 UTC) from {username='slave'} at 127.0.0.1:50754
I20260928 04:47:07.422155 21026 raft_consensus.cc:2258] T b773659dc0cd4d2b9c9c573e8d831e44 P 35e24b8388fe4d83be91a8843f0fc883 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:47:07.422269 21027 tablet_replica.cc:330] T b773659dc0cd4d2b9c9c573e8d831e44 P d55c397758ce4f91a83c2f59446b0392: stopping tablet replica
I20260928 04:47:07.422387 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 20599
I20260928 04:47:07.422523 21027 raft_consensus.cc:2229] T b773659dc0cd4d2b9c9c573e8d831e44 P d55c397758ce4f91a83c2f59446b0392 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:47:07.422631 21026 ts_tablet_manager.cc:1905] T b773659dc0cd4d2b9c9c573e8d831e44 P 35e24b8388fe4d83be91a8843f0fc883: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:47:07.422739 21027 raft_consensus.cc:2258] T b773659dc0cd4d2b9c9c573e8d831e44 P d55c397758ce4f91a83c2f59446b0392 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:47:07.424715 21026 ts_tablet_manager.cc:1918] T b773659dc0cd4d2b9c9c573e8d831e44 P 35e24b8388fe4d83be91a8843f0fc883: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:47:07.424788 21026 log.cc:1196] T b773659dc0cd4d2b9c9c573e8d831e44 P 35e24b8388fe4d83be91a8843f0fc883: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-2/wal/wals/b773659dc0cd4d2b9c9c573e8d831e44
I20260928 04:47:07.425045 21026 ts_tablet_manager.cc:1939] T b773659dc0cd4d2b9c9c573e8d831e44 P 35e24b8388fe4d83be91a8843f0fc883: Deleting consensus metadata
I20260928 04:47:07.425488 20525 catalog_manager.cc:4841] TS 35e24b8388fe4d83be91a8843f0fc883 (127.1.226.195:37241): tablet b773659dc0cd4d2b9c9c573e8d831e44 (table test-table [id=b8faea945a4447d2ae6c6ff25654d8f4]) successfully deleted
I20260928 04:47:07.426313 21025 tablet_replica.cc:330] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17: stopping tablet replica
I20260928 04:47:07.426470 21025 raft_consensus.cc:2229] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:47:07.426625 21025 raft_consensus.cc:2258] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:47:07.427134 21025 ts_tablet_manager.cc:1905] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:47:07.428755 21025 ts_tablet_manager.cc:1918] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:47:07.428885 21025 log.cc:1196] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570585693787-1931-0/minicluster-data/ts-1/wal/wals/b773659dc0cd4d2b9c9c573e8d831e44
W20260928 04:47:07.428978 20525 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.226.193:46151 (error 108)
I20260928 04:47:07.429237 21025 ts_tablet_manager.cc:1939] T b773659dc0cd4d2b9c9c573e8d831e44 P 696c4d9cc2584f089c6c0805f97b8a17: Deleting consensus metadata
I20260928 04:47:07.429250 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 20730
I20260928 04:47:07.430410 20998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:07.430275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:46151 (local address 127.0.0.1:50760)
0928 04:47:07.430334 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:07.430374 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.193:46151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 04:47:07.430522 20525 catalog_manager.cc:4584] TS d55c397758ce4f91a83c2f59446b0392 (127.1.226.193:46151): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b773659dc0cd4d2b9c9c573e8d831e44: Network error: Client connection negotiation failed: client connection to 127.1.226.193:46151: connect: Connection refused (error 111)
I20260928 04:47:07.434367 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 20861
I20260928 04:47:07.434402 20998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:07.434279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:40319 (local address 127.0.0.1:59180)
0928 04:47:07.434331 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:07.434364 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.194:40319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 04:47:07.439208 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 20508
2026-09-28T04:47:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (867 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:47:07 dist-test-slave-j5c3 krb5kdc[21031](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:47:07 dist-test-slave-j5c3 krb5kdc[21031](info): set up 2 sockets
Sep 28 04:47:07 dist-test-slave-j5c3 krb5kdc[21031](info): commencing operation
krb5kdc: starting...
W20260928 04:47:08.480595 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:47:08 dist-test-slave-j5c3 krb5kdc[21031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:47:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:47:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:47:08.603885 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:42359
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:46863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:42359
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:08.690975 21044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:08.691191 21044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:08.691228 21044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:08.692416 21044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:08.692457 21044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:08.692472 21044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:08.692484 21044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:08.692497 21044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:08.693936 21044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:46863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:42359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:42359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:08.694133 21044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:08.694314 21044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:08.699204 21052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:08.699201 21049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:08.699241 21050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:08.699669 21044 server_base.cc:1034] running on GCE node
I20260928 04:47:08.699910 21044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:08.700202 21044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:08.701377 21044 hybrid_clock.cc:648] HybridClock initialized: now 1790570828701370 us; error 33 us; skew 500 ppm
Sep 28 04:47:08 dist-test-slave-j5c3 krb5kdc[21031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570828, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:08.704041 21044 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:47:08.704584 21044 webserver.cc:460] Webserver started at http://127.1.226.254:41985/ using document root <none> and password file <none>
I20260928 04:47:08.704818 21044 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:08.704887 21044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:08.705024 21044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:08.706097 21044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "7b8e901a6ff24b81a573780a9c96b135"
format_stamp: "Formatted at 2026-09-28 04:47:08 on dist-test-slave-j5c3"
server_key: "803267f58aa79d45bc12d22af61e4894"
server_key_iv: "050bc7258ed8de98392c6ee4b24a4960"
server_key_version: "encryptionkey@0"
I20260928 04:47:08.706480 21044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "7b8e901a6ff24b81a573780a9c96b135"
format_stamp: "Formatted at 2026-09-28 04:47:08 on dist-test-slave-j5c3"
server_key: "803267f58aa79d45bc12d22af61e4894"
server_key_iv: "050bc7258ed8de98392c6ee4b24a4960"
server_key_version: "encryptionkey@0"
I20260928 04:47:08.707907 21044 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:47:08.708873 21059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:08.709060 21044 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:08.709180 21044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "7b8e901a6ff24b81a573780a9c96b135"
format_stamp: "Formatted at 2026-09-28 04:47:08 on dist-test-slave-j5c3"
server_key: "803267f58aa79d45bc12d22af61e4894"
server_key_iv: "050bc7258ed8de98392c6ee4b24a4960"
server_key_version: "encryptionkey@0"
I20260928 04:47:08.709280 21044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:08.735723 21044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:08.738313 21044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:08.738492 21044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:08.742827 21044 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:42359
I20260928 04:47:08.742895 21111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:42359 every 8 connection(s)
I20260928 04:47:08.743271 21044 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:47:08.744280 21112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:08.746735 21112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b8e901a6ff24b81a573780a9c96b135: Bootstrap starting.
I20260928 04:47:08.747331 21112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b8e901a6ff24b81a573780a9c96b135: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:08.747583 21112 log.cc:824] T 00000000000000000000000000000000 P 7b8e901a6ff24b81a573780a9c96b135: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:08.748385 21112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b8e901a6ff24b81a573780a9c96b135: No bootstrap required, opened a new log
I20260928 04:47:08.749305 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 21044
I20260928 04:47:08.749451 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:47:08.749634 1931 external_mini_cluster.cc:1428] Setting key aa184ddfa08db76f9638f800dc3462be
I20260928 04:47:08.750689 21112 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b8e901a6ff24b81a573780a9c96b135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8e901a6ff24b81a573780a9c96b135" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42359 } }
I20260928 04:47:08.750893 21112 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b8e901a6ff24b81a573780a9c96b135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:08.750923 21112 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b8e901a6ff24b81a573780a9c96b135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8e901a6ff24b81a573780a9c96b135, State: Initialized, Role: FOLLOWER
I20260928 04:47:08.751096 21112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b8e901a6ff24b81a573780a9c96b135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8e901a6ff24b81a573780a9c96b135" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42359 } }
I20260928 04:47:08.751205 21112 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b8e901a6ff24b81a573780a9c96b135 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:47:08.751255 21112 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b8e901a6ff24b81a573780a9c96b135 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:47:08.751314 21112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b8e901a6ff24b81a573780a9c96b135 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:08.752431 21112 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b8e901a6ff24b81a573780a9c96b135 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8e901a6ff24b81a573780a9c96b135" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42359 } }
I20260928 04:47:08.752565 21112 leader_election.cc:302] T 00000000000000000000000000000000 P 7b8e901a6ff24b81a573780a9c96b135 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b8e901a6ff24b81a573780a9c96b135; no voters:
I20260928 04:47:08.752790 21112 leader_election.cc:288] T 00000000000000000000000000000000 P 7b8e901a6ff24b81a573780a9c96b135 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:47:08.752810 21117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b8e901a6ff24b81a573780a9c96b135 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:08.752978 21117 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b8e901a6ff24b81a573780a9c96b135 [term 1 LEADER]: Becoming Leader. State: Replica: 7b8e901a6ff24b81a573780a9c96b135, State: Running, Role: LEADER
I20260928 04:47:08.753093 21112 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b8e901a6ff24b81a573780a9c96b135 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:08.753187 21117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b8e901a6ff24b81a573780a9c96b135 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8e901a6ff24b81a573780a9c96b135" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42359 } }
I20260928 04:47:08.753643 21117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b8e901a6ff24b81a573780a9c96b135 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b8e901a6ff24b81a573780a9c96b135. Latest consensus state: current_term: 1 leader_uuid: "7b8e901a6ff24b81a573780a9c96b135" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8e901a6ff24b81a573780a9c96b135" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42359 } } }
I20260928 04:47:08.753787 21117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b8e901a6ff24b81a573780a9c96b135 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:08.754134 21124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:47:08.754595 21119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b8e901a6ff24b81a573780a9c96b135 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b8e901a6ff24b81a573780a9c96b135" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8e901a6ff24b81a573780a9c96b135" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 42359 } } }
I20260928 04:47:08.754721 21119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b8e901a6ff24b81a573780a9c96b135 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:08.755062 21124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:47:08.756958 21124 catalog_manager.cc:1347] Generated new cluster ID: ce40c60dbf864ebbb64de720bf51ca9e
I20260928 04:47:08.757030 21124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:47:08 dist-test-slave-j5c3 krb5kdc[21031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:47:08.763674 21116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:08.750224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33014 (local address 127.1.226.254:42359)
0928 04:47:08.750432 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:08.750444 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:08.750711 (+ 267us) server_negotiation.cc:407] Connection header received
0928 04:47:08.750829 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:08.750835 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:08.750883 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:08.751079 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:08.751824 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:08.753322 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:08.754644 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:08.755523 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:08.755647 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:08.759148 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:08.759152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:08.759153 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:08.759168 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:08.761669 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:08.762146 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:08.762149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:08.762150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:08.762300 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:08.762591 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:08.762591 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:08.762592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:08.762964 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:08.763139 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:08.763270 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 04:47:08.763427 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":82,"threads_started":1}
I20260928 04:47:08.779563 21124 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:47:08.780243 21124 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:47:08.786825 21124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b8e901a6ff24b81a573780a9c96b135: Generated new TSK 0
I20260928 04:47:08.787058 21124 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.226.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
Entry for principal kudu/127.1.226.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab.
I20260928 04:47:08.815236 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:0
--local_ip_for_outbound_sockets=127.1.226.193
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:42359
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:46863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:08.900565 21138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:08.900828 21138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:08.900869 21138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:08.902084 21138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:08.902155 21138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:08.902199 21138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:47:08.903705 21138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:46863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.193.keytab
--principal=kudu/127.1.226.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:42359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:08.903996 21138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:08.904270 21138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:08.909348 21143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:08.909358 21146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:08.909358 21144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:08.909511 21138 server_base.cc:1034] running on GCE node
I20260928 04:47:08.909788 21138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:08.910072 21138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:08.911273 21138 hybrid_clock.cc:648] HybridClock initialized: now 1790570828911260 us; error 31 us; skew 500 ppm
Sep 28 04:47:08 dist-test-slave-j5c3 krb5kdc[21031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570828, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:08.914065 21138 init.cc:377] Logged in from keytab as kudu/127.1.226.193@KRBTEST.COM (short username kudu)
I20260928 04:47:08.914469 21138 webserver.cc:460] Webserver started at http://127.1.226.193:42341/ using document root <none> and password file <none>
I20260928 04:47:08.914667 21138 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:08.914718 21138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:08.914808 21138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:08.915872 21138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-0/data/instance:
uuid: "227003c258b549cdbfc248d7b4c3b7fa"
format_stamp: "Formatted at 2026-09-28 04:47:08 on dist-test-slave-j5c3"
server_key: "50c99431bdb53c1feb9ac577fdb3c72d"
server_key_iv: "e504ad2626206ef68993d170bafe5e98"
server_key_version: "encryptionkey@0"
I20260928 04:47:08.916253 21138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance:
uuid: "227003c258b549cdbfc248d7b4c3b7fa"
format_stamp: "Formatted at 2026-09-28 04:47:08 on dist-test-slave-j5c3"
server_key: "50c99431bdb53c1feb9ac577fdb3c72d"
server_key_iv: "e504ad2626206ef68993d170bafe5e98"
server_key_version: "encryptionkey@0"
I20260928 04:47:08.917591 21138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:47:08.918442 21153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:08.918635 21138 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:47:08.918722 21138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "227003c258b549cdbfc248d7b4c3b7fa"
format_stamp: "Formatted at 2026-09-28 04:47:08 on dist-test-slave-j5c3"
server_key: "50c99431bdb53c1feb9ac577fdb3c72d"
server_key_iv: "e504ad2626206ef68993d170bafe5e98"
server_key_version: "encryptionkey@0"
I20260928 04:47:08.918803 21138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:08.933243 21138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:08.935876 21138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:08.936012 21138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:08.936311 21138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:47:08.936651 21138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:47:08.936694 21138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:08.936718 21138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:47:08.936731 21138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:08.942906 21138 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:43573
I20260928 04:47:08.943015 21266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:43573 every 8 connection(s)
I20260928 04:47:08.943375 21138 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
Sep 28 04:47:08 dist-test-slave-j5c3 krb5kdc[21031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570828, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.193@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:47:08.951020 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 21138
I20260928 04:47:08.951169 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance
I20260928 04:47:08.951434 1931 external_mini_cluster.cc:1428] Setting key 7ae3be1b979f1635c1b0ef5dd799ed07
I20260928 04:47:08.955307 21116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:08.944300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:58681 (local address 127.1.226.254:42359)
0928 04:47:08.944395 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:08.944397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:08.945680 (+ 1283us) server_negotiation.cc:407] Connection header received
0928 04:47:08.946152 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:08.946153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:08.946165 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:08.946226 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:08.947204 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:08.947732 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:08.948842 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:08.949060 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:08.949144 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:08.951610 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:08.951625 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:08.951626 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:08.951636 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:08.953524 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:08.954068 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:08.954069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:08.954069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:08.954141 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:08.954440 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:08.954441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:08.954441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:08.954601 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:08.954679 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:08.955049 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 04:47:08.955156 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 04:47:08.955351 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:08.944299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42359 (local address 127.1.226.193:58681)
0928 04:47:08.945514 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:08.945572 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:08.946062 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:08.946283 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:08.946286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:08.946879 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:08.947082 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:08.947087 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:08.947841 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:08.947842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:08.948742 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:08.948745 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:08.949153 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:08.949154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:08.949258 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:08.949869 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:47:08.949877 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:08.951491 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:08.953621 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:08.953632 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:08.953636 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:08.953994 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:08.954227 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:08.954228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:08.954229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:08.954369 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:08.954675 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:08.954680 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:08.954888 (+ 208us) client_negotiation.cc:769] Sending connection context
0928 04:47:08.955018 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 04:47:08.955165 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1135,"thread_start_us":101,"threads_started":1}
I20260928 04:47:08.956013 21267 heartbeater.cc:347] Connected to a master server at 127.1.226.254:42359
I20260928 04:47:08.956141 21267 heartbeater.cc:464] Registering TS with master...
I20260928 04:47:08.956409 21267 heartbeater.cc:511] Master 127.1.226.254:42359 requested a full tablet report, sending...
I20260928 04:47:08.957130 21076 ts_manager.cc:194] Registered new tserver with Master: 227003c258b549cdbfc248d7b4c3b7fa (127.1.226.193:43573)
I20260928 04:47:08.958143 21076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.193@KRBTEST.COM'} at 127.1.226.193:58681
WARNING: no policy specified for kudu/127.1.226.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
Entry for principal kudu/127.1.226.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab.
I20260928 04:47:08.978874 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:0
--local_ip_for_outbound_sockets=127.1.226.194
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:42359
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:46863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:09.063503 21272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:09.063709 21272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:09.063740 21272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:09.064929 21272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:09.064968 21272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:09.064994 21272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:47:09.066429 21272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:46863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.194.keytab
--principal=kudu/127.1.226.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:42359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:09.066684 21272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:09.066943 21272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:09.072212 21277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:09.072201 21280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:09.072201 21278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:09.072413 21272 server_base.cc:1034] running on GCE node
I20260928 04:47:09.072647 21272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:09.072922 21272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:09.074098 21272 hybrid_clock.cc:648] HybridClock initialized: now 1790570829074081 us; error 39 us; skew 500 ppm
Sep 28 04:47:09 dist-test-slave-j5c3 krb5kdc[21031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570829, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:09.076813 21272 init.cc:377] Logged in from keytab as kudu/127.1.226.194@KRBTEST.COM (short username kudu)
I20260928 04:47:09.077339 21272 webserver.cc:460] Webserver started at http://127.1.226.194:45937/ using document root <none> and password file <none>
I20260928 04:47:09.077543 21272 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:09.077584 21272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:09.077674 21272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:09.078688 21272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-1/data/instance:
uuid: "cf9ecdb609f94cf4a215f367199a4903"
format_stamp: "Formatted at 2026-09-28 04:47:09 on dist-test-slave-j5c3"
server_key: "fd83b9f860327b3b412981078efdcf26"
server_key_iv: "9629bf3cd3f5391c906c96b2f015b64a"
server_key_version: "encryptionkey@0"
I20260928 04:47:09.078995 21272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance:
uuid: "cf9ecdb609f94cf4a215f367199a4903"
format_stamp: "Formatted at 2026-09-28 04:47:09 on dist-test-slave-j5c3"
server_key: "fd83b9f860327b3b412981078efdcf26"
server_key_iv: "9629bf3cd3f5391c906c96b2f015b64a"
server_key_version: "encryptionkey@0"
I20260928 04:47:09.080343 21272 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:47:09.081095 21287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:09.081331 21272 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:09.081403 21272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "cf9ecdb609f94cf4a215f367199a4903"
format_stamp: "Formatted at 2026-09-28 04:47:09 on dist-test-slave-j5c3"
server_key: "fd83b9f860327b3b412981078efdcf26"
server_key_iv: "9629bf3cd3f5391c906c96b2f015b64a"
server_key_version: "encryptionkey@0"
I20260928 04:47:09.081493 21272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:09.107481 21272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:09.109979 21272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:09.110136 21272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:09.110399 21272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:47:09.110723 21272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:47:09.110759 21272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:09.111135 21272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:47:09.111231 21272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:09.117623 21272 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:35799
I20260928 04:47:09.117682 21400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:35799 every 8 connection(s)
I20260928 04:47:09.118062 21272 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
I20260928 04:47:09.124522 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 21272
I20260928 04:47:09.124671 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance
I20260928 04:47:09.124971 1931 external_mini_cluster.cc:1428] Setting key d7a993d24a1851116b03ab2da4d7e50c
Sep 28 04:47:09 dist-test-slave-j5c3 krb5kdc[21031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570829, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.194@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:47:09.129899 21403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:09.118899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42359 (local address 127.1.226.194:58695)
0928 04:47:09.119098 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:09.119137 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:09.119663 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:09.120027 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:09.120030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:09.120689 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:09.120894 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:09.120899 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.121730 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.121730 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:09.122630 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:09.122633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.123216 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.123217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:09.123350 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:09.123945 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:47:09.123954 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:09.125837 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:09.127953 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:09.127962 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:09.127965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:09.128382 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:09.128661 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:09.128663 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:09.128663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:09.128908 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:09.129262 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:09.129269 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:09.129464 (+ 195us) client_negotiation.cc:769] Sending connection context
0928 04:47:09.129562 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 04:47:09.129717 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":108,"threads_started":1}
I20260928 04:47:09.129957 21116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:09.118928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:58695 (local address 127.1.226.254:42359)
0928 04:47:09.118978 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:09.118980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:09.119229 (+ 249us) server_negotiation.cc:407] Connection header received
0928 04:47:09.119848 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:09.119849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:09.119861 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:09.119941 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:09.121047 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.121568 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.122750 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.123084 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.123159 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:09.125972 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:09.125974 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:09.125975 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:09.125983 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:09.127842 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:09.128500 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:09.128501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:09.128501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:09.128570 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:09.129012 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:09.129012 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:09.129013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:09.129171 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:09.129245 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:09.129627 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 04:47:09.129810 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:47:09.130512 21401 heartbeater.cc:347] Connected to a master server at 127.1.226.254:42359
I20260928 04:47:09.130724 21401 heartbeater.cc:464] Registering TS with master...
I20260928 04:47:09.130950 21401 heartbeater.cc:511] Master 127.1.226.254:42359 requested a full tablet report, sending...
I20260928 04:47:09.131430 21076 ts_manager.cc:194] Registered new tserver with Master: cf9ecdb609f94cf4a215f367199a4903 (127.1.226.194:35799)
I20260928 04:47:09.132015 21076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.194@KRBTEST.COM'} at 127.1.226.194:58695
WARNING: no policy specified for kudu/127.1.226.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
Entry for principal kudu/127.1.226.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab.
I20260928 04:47:09.155607 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:0
--local_ip_for_outbound_sockets=127.1.226.195
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:42359
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:46863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:09.241483 21406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:09.241688 21406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:09.241719 21406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:09.242884 21406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:09.242923 21406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:09.242949 21406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:47:09.244467 21406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:46863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.195.keytab
--principal=kudu/127.1.226.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:42359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:09.244699 21406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:09.244938 21406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:09.250068 21412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:09.250111 21411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:09.250111 21414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:09.250577 21406 server_base.cc:1034] running on GCE node
I20260928 04:47:09.250835 21406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:09.251075 21406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:09.252224 21406 hybrid_clock.cc:648] HybridClock initialized: now 1790570829252211 us; error 41 us; skew 500 ppm
Sep 28 04:47:09 dist-test-slave-j5c3 krb5kdc[21031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570829, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:09.255185 21406 init.cc:377] Logged in from keytab as kudu/127.1.226.195@KRBTEST.COM (short username kudu)
I20260928 04:47:09.255679 21406 webserver.cc:460] Webserver started at http://127.1.226.195:36133/ using document root <none> and password file <none>
I20260928 04:47:09.255868 21406 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:09.255914 21406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:09.256008 21406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:09.257148 21406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-2/data/instance:
uuid: "46843b624ba8474e9261e232bd7ab072"
format_stamp: "Formatted at 2026-09-28 04:47:09 on dist-test-slave-j5c3"
server_key: "a4f8f37860e0e40920541247c0eedc64"
server_key_iv: "a7bfab2547f81937ade8ee7d22a5df9a"
server_key_version: "encryptionkey@0"
I20260928 04:47:09.257486 21406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance:
uuid: "46843b624ba8474e9261e232bd7ab072"
format_stamp: "Formatted at 2026-09-28 04:47:09 on dist-test-slave-j5c3"
server_key: "a4f8f37860e0e40920541247c0eedc64"
server_key_iv: "a7bfab2547f81937ade8ee7d22a5df9a"
server_key_version: "encryptionkey@0"
I20260928 04:47:09.258855 21406 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:47:09.259606 21421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:09.259871 21406 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:09.259979 21406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "46843b624ba8474e9261e232bd7ab072"
format_stamp: "Formatted at 2026-09-28 04:47:09 on dist-test-slave-j5c3"
server_key: "a4f8f37860e0e40920541247c0eedc64"
server_key_iv: "a7bfab2547f81937ade8ee7d22a5df9a"
server_key_version: "encryptionkey@0"
I20260928 04:47:09.260056 21406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:09.274369 21406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:09.276412 21406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:09.276568 21406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:09.276818 21406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:47:09.277158 21406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:47:09.277190 21406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:09.277235 21406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:47:09.277262 21406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:09.283066 21406 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:37133
I20260928 04:47:09.283115 21534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:37133 every 8 connection(s)
I20260928 04:47:09.283427 21406 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
Sep 28 04:47:09 dist-test-slave-j5c3 krb5kdc[21031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570829, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.195@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:47:09.291054 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 21406
I20260928 04:47:09.291185 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance
I20260928 04:47:09.291419 1931 external_mini_cluster.cc:1428] Setting key 8ed2d9524acace230a7e386deac4f64e
I20260928 04:47:09.293674 21116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:09.284143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:56265 (local address 127.1.226.254:42359)
0928 04:47:09.284253 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:09.284255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:09.284430 (+ 175us) server_negotiation.cc:407] Connection header received
0928 04:47:09.284942 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:09.284944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:09.284956 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:09.285014 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:09.286039 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.286573 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.287771 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.287987 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.288068 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:09.290440 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:09.290443 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:09.290443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:09.290451 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:09.292099 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:09.292596 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:09.292598 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:09.292598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:09.292676 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:09.292964 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:09.292965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:09.292965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:09.293092 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:09.293167 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:09.293492 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 04:47:09.293580 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 04:47:09.293934 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:09.284136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:42359 (local address 127.1.226.195:56265)
0928 04:47:09.284315 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:09.284357 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:09.284832 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:09.285091 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:09.285095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:09.285741 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:09.285937 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:09.285942 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.286691 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.286693 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:09.287657 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:09.287660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.288075 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.288076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:09.288279 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:09.288819 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:47:09.288827 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:09.290335 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:09.292198 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:09.292205 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:09.292207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:09.292497 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:09.292763 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:09.292764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:09.292764 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:09.292890 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:09.293168 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:09.293176 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:09.293345 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 04:47:09.293458 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 04:47:09.293631 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":111,"threads_started":1}
I20260928 04:47:09.294435 21535 heartbeater.cc:347] Connected to a master server at 127.1.226.254:42359
I20260928 04:47:09.294518 21535 heartbeater.cc:464] Registering TS with master...
I20260928 04:47:09.294714 21535 heartbeater.cc:511] Master 127.1.226.254:42359 requested a full tablet report, sending...
I20260928 04:47:09.295100 21076 ts_manager.cc:194] Registered new tserver with Master: 46843b624ba8474e9261e232bd7ab072 (127.1.226.195:37133)
I20260928 04:47:09.295501 21076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.226.195@KRBTEST.COM'} at 127.1.226.195:56265
I20260928 04:47:09.304327 1931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:47:09.312261 21116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:09.305854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33020 (local address 127.1.226.254:42359)
0928 04:47:09.305927 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:09.305929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:09.306015 (+ 86us) server_negotiation.cc:407] Connection header received
0928 04:47:09.306049 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:09.306050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:09.306062 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:09.306129 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:09.306687 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.307204 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.307912 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.308098 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.308258 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:09.309454 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:09.309455 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:09.309455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:09.309463 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:09.310894 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:09.311306 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:09.311307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:09.311307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:09.311361 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:09.311568 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:09.311569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:09.311569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:09.311788 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:09.311863 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:09.312045 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 04:47:09.312106 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 04:47:09.314209 21076 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:47:09.315920 21076 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:47:09.323155 21547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:09.318597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:35799 (local address 127.0.0.1:57474)
0928 04:47:09.318785 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:09.318796 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:09.318853 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:09.319320 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:09.319321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:09.319323 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:09.319430 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:09.319432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.320821 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.320822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:09.321925 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:09.321928 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.322805 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.322806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:09.322887 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:09.322910 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 04:47:09.323010 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 04:47:09.323069 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":12,"thread_start_us":119,"threads_started":1}
I20260928 04:47:09.323160 21550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:09.318953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57474 (local address 127.1.226.194:35799)
0928 04:47:09.319129 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:09.319134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:09.319144 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:47:09.319202 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:09.319204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:09.319221 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:09.319312 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:09.319531 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.320690 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.322021 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.322699 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.322802 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:09.322868 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:09.323030 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 04:47:09.323075 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":100,"threads_started":1}
I20260928 04:47:09.323362 21546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:09.318560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:37133 (local address 127.0.0.1:45076)
0928 04:47:09.318672 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:09.318716 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:09.318841 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:09.319189 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:09.319192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:09.319195 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:09.319305 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:09.319308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.320836 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.320836 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:09.322156 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:09.322159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.323100 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.323101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:09.323174 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:09.323178 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:47:09.323208 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 04:47:09.323271 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":109,"thread_start_us":60,"threads_started":1}
I20260928 04:47:09.323524 21551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:09.319246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57172 (local address 127.1.226.193:43573)
0928 04:47:09.319624 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:09.319629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:09.319638 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:47:09.319676 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:09.319678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:09.319693 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:09.319759 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:09.319893 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.320673 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.322258 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.322989 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.323064 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:09.323121 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:09.323398 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 04:47:09.323448 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":82,"threads_started":1}
I20260928 04:47:09.323362 21548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:09.318780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:43573 (local address 127.0.0.1:57172)
0928 04:47:09.319418 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:09.319428 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:09.319473 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:09.319776 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:09.319777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:09.319779 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:09.319832 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:09.319834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.320779 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.320780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:09.322173 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:09.322175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.323149 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.323149 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:09.323205 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:09.323208 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:47:09.323253 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 04:47:09.323281 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"mutex_wait_us":63,"thread_start_us":90,"threads_started":1}
I20260928 04:47:09.323676 21549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:09.318796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45076 (local address 127.1.226.195:37133)
0928 04:47:09.319010 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:09.319014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:09.319023 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:47:09.319057 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:09.319058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:09.319072 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:09.319185 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:09.319411 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.320693 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.322271 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.323007 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.323446 (+ 439us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:09.323507 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:09.323533 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 04:47:09.323577 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":98,"threads_started":1}
I20260928 04:47:09.324472 21201 tablet_service.cc:1467] Processing CreateTablet for tablet 963bde0f89244407b95cf4653e1016c9 (DEFAULT_TABLE table=test-table [id=5f14f99096504dfd8cdcc90e72ac539e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:47:09.324527 21469 tablet_service.cc:1467] Processing CreateTablet for tablet 963bde0f89244407b95cf4653e1016c9 (DEFAULT_TABLE table=test-table [id=5f14f99096504dfd8cdcc90e72ac539e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:47:09.324836 21201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 963bde0f89244407b95cf4653e1016c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:09.324894 21469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 963bde0f89244407b95cf4653e1016c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:09.327579 21552 tablet_bootstrap.cc:492] T 963bde0f89244407b95cf4653e1016c9 P 227003c258b549cdbfc248d7b4c3b7fa: Bootstrap starting.
I20260928 04:47:09.327739 21553 tablet_bootstrap.cc:492] T 963bde0f89244407b95cf4653e1016c9 P 46843b624ba8474e9261e232bd7ab072: Bootstrap starting.
I20260928 04:47:09.328058 21335 tablet_service.cc:1467] Processing CreateTablet for tablet 963bde0f89244407b95cf4653e1016c9 (DEFAULT_TABLE table=test-table [id=5f14f99096504dfd8cdcc90e72ac539e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:47:09.328254 21552 tablet_bootstrap.cc:654] T 963bde0f89244407b95cf4653e1016c9 P 227003c258b549cdbfc248d7b4c3b7fa: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:09.328449 21335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 963bde0f89244407b95cf4653e1016c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:09.328583 21553 tablet_bootstrap.cc:654] T 963bde0f89244407b95cf4653e1016c9 P 46843b624ba8474e9261e232bd7ab072: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:09.328794 21552 log.cc:824] T 963bde0f89244407b95cf4653e1016c9 P 227003c258b549cdbfc248d7b4c3b7fa: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:09.329123 21553 log.cc:824] T 963bde0f89244407b95cf4653e1016c9 P 46843b624ba8474e9261e232bd7ab072: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:09.329628 21552 tablet_bootstrap.cc:492] T 963bde0f89244407b95cf4653e1016c9 P 227003c258b549cdbfc248d7b4c3b7fa: No bootstrap required, opened a new log
I20260928 04:47:09.329697 21552 ts_tablet_manager.cc:1397] T 963bde0f89244407b95cf4653e1016c9 P 227003c258b549cdbfc248d7b4c3b7fa: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 04:47:09.329972 21553 tablet_bootstrap.cc:492] T 963bde0f89244407b95cf4653e1016c9 P 46843b624ba8474e9261e232bd7ab072: No bootstrap required, opened a new log
I20260928 04:47:09.330056 21553 ts_tablet_manager.cc:1397] T 963bde0f89244407b95cf4653e1016c9 P 46843b624ba8474e9261e232bd7ab072: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 04:47:09.331384 21552 raft_consensus.cc:348] T 963bde0f89244407b95cf4653e1016c9 P 227003c258b549cdbfc248d7b4c3b7fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9ecdb609f94cf4a215f367199a4903" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35799 } } peers { permanent_uuid: "46843b624ba8474e9261e232bd7ab072" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 37133 } } peers { permanent_uuid: "227003c258b549cdbfc248d7b4c3b7fa" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43573 } }
I20260928 04:47:09.331548 21552 raft_consensus.cc:374] T 963bde0f89244407b95cf4653e1016c9 P 227003c258b549cdbfc248d7b4c3b7fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:09.331591 21552 raft_consensus.cc:729] T 963bde0f89244407b95cf4653e1016c9 P 227003c258b549cdbfc248d7b4c3b7fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 227003c258b549cdbfc248d7b4c3b7fa, State: Initialized, Role: FOLLOWER
I20260928 04:47:09.331707 21552 consensus_queue.cc:260] T 963bde0f89244407b95cf4653e1016c9 P 227003c258b549cdbfc248d7b4c3b7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9ecdb609f94cf4a215f367199a4903" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35799 } } peers { permanent_uuid: "46843b624ba8474e9261e232bd7ab072" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 37133 } } peers { permanent_uuid: "227003c258b549cdbfc248d7b4c3b7fa" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43573 } }
I20260928 04:47:09.332000 21552 ts_tablet_manager.cc:1428] T 963bde0f89244407b95cf4653e1016c9 P 227003c258b549cdbfc248d7b4c3b7fa: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 04:47:09.332026 21267 heartbeater.cc:503] Master 127.1.226.254:42359 was elected leader, sending a full tablet report...
I20260928 04:47:09.332186 21557 tablet_bootstrap.cc:492] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903: Bootstrap starting.
I20260928 04:47:09.332914 21557 tablet_bootstrap.cc:654] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:09.332969 21553 raft_consensus.cc:348] T 963bde0f89244407b95cf4653e1016c9 P 46843b624ba8474e9261e232bd7ab072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9ecdb609f94cf4a215f367199a4903" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35799 } } peers { permanent_uuid: "46843b624ba8474e9261e232bd7ab072" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 37133 } } peers { permanent_uuid: "227003c258b549cdbfc248d7b4c3b7fa" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43573 } }
I20260928 04:47:09.333192 21553 raft_consensus.cc:374] T 963bde0f89244407b95cf4653e1016c9 P 46843b624ba8474e9261e232bd7ab072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:09.333248 21553 raft_consensus.cc:729] T 963bde0f89244407b95cf4653e1016c9 P 46843b624ba8474e9261e232bd7ab072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46843b624ba8474e9261e232bd7ab072, State: Initialized, Role: FOLLOWER
I20260928 04:47:09.333432 21553 consensus_queue.cc:260] T 963bde0f89244407b95cf4653e1016c9 P 46843b624ba8474e9261e232bd7ab072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9ecdb609f94cf4a215f367199a4903" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35799 } } peers { permanent_uuid: "46843b624ba8474e9261e232bd7ab072" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 37133 } } peers { permanent_uuid: "227003c258b549cdbfc248d7b4c3b7fa" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43573 } }
I20260928 04:47:09.333503 21557 log.cc:824] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:09.333750 21553 ts_tablet_manager.cc:1428] T 963bde0f89244407b95cf4653e1016c9 P 46843b624ba8474e9261e232bd7ab072: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 04:47:09.333853 21535 heartbeater.cc:503] Master 127.1.226.254:42359 was elected leader, sending a full tablet report...
I20260928 04:47:09.334437 21557 tablet_bootstrap.cc:492] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903: No bootstrap required, opened a new log
I20260928 04:47:09.334528 21557 ts_tablet_manager.cc:1397] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:47:09.336095 21557 raft_consensus.cc:348] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9ecdb609f94cf4a215f367199a4903" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35799 } } peers { permanent_uuid: "46843b624ba8474e9261e232bd7ab072" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 37133 } } peers { permanent_uuid: "227003c258b549cdbfc248d7b4c3b7fa" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43573 } }
I20260928 04:47:09.336316 21557 raft_consensus.cc:374] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:09.336367 21557 raft_consensus.cc:729] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf9ecdb609f94cf4a215f367199a4903, State: Initialized, Role: FOLLOWER
I20260928 04:47:09.336481 21557 consensus_queue.cc:260] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9ecdb609f94cf4a215f367199a4903" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35799 } } peers { permanent_uuid: "46843b624ba8474e9261e232bd7ab072" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 37133 } } peers { permanent_uuid: "227003c258b549cdbfc248d7b4c3b7fa" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43573 } }
I20260928 04:47:09.336742 21557 ts_tablet_manager.cc:1428] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 04:47:09.336812 21401 heartbeater.cc:503] Master 127.1.226.254:42359 was elected leader, sending a full tablet report...
W20260928 04:47:09.368669 21402 tablet.cc:2369] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:47:09.427214 21560 raft_consensus.cc:482] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:47:09.427350 21560 raft_consensus.cc:504] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9ecdb609f94cf4a215f367199a4903" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35799 } } peers { permanent_uuid: "46843b624ba8474e9261e232bd7ab072" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 37133 } } peers { permanent_uuid: "227003c258b549cdbfc248d7b4c3b7fa" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43573 } }
I20260928 04:47:09.427678 21560 leader_election.cc:288] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46843b624ba8474e9261e232bd7ab072 (127.1.226.195:37133), 227003c258b549cdbfc248d7b4c3b7fa (127.1.226.193:43573)
I20260928 04:47:09.430881 21551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:09.428249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:43371 (local address 127.1.226.193:43573)
0928 04:47:09.428305 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:09.428307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:09.428433 (+ 126us) server_negotiation.cc:407] Connection header received
0928 04:47:09.428503 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:09.428504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:09.428513 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:09.428558 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:09.428721 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.429192 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.430134 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.430504 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.430601 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:09.430610 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:09.430756 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 04:47:09.430799 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 04:47:09.430881 21549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:09.428142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:49641 (local address 127.1.226.195:37133)
0928 04:47:09.428214 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:09.428217 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:09.428229 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:47:09.428334 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:09.428335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:09.428344 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:09.428420 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:09.428586 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.429078 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.430090 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.430504 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.430593 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:09.430603 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:09.430766 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 04:47:09.430806 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 04:47:09.430888 21403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:09.428012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:37133 (local address 127.1.226.194:49641)
0928 04:47:09.428108 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:09.428123 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:09.428239 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:09.428424 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:09.428425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:09.428426 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:09.428504 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:09.428506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.429170 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.429172 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:09.429989 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:09.429991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.430590 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.430591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:09.430675 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:09.430680 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:47:09.430745 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 04:47:09.430791 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":1}
I20260928 04:47:09.430896 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:09.428135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:43573 (local address 127.1.226.194:43371)
0928 04:47:09.428340 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:09.428358 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:09.428437 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:09.428569 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:09.428570 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:09.428571 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:09.428643 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:09.428645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.429286 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.429287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:09.430038 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:09.430040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.430602 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.430603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:09.430675 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:09.430680 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:47:09.430750 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 04:47:09.430793 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":142,"threads_started":1}
I20260928 04:47:09.431352 21489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "963bde0f89244407b95cf4653e1016c9" candidate_uuid: "cf9ecdb609f94cf4a215f367199a4903" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46843b624ba8474e9261e232bd7ab072" is_pre_election: true
I20260928 04:47:09.431509 21489 raft_consensus.cc:2454] T 963bde0f89244407b95cf4653e1016c9 P 46843b624ba8474e9261e232bd7ab072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf9ecdb609f94cf4a215f367199a4903 in term 0.
I20260928 04:47:09.431798 21291 leader_election.cc:302] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46843b624ba8474e9261e232bd7ab072, cf9ecdb609f94cf4a215f367199a4903; no voters:
I20260928 04:47:09.431947 21560 raft_consensus.cc:2781] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:47:09.431915 21221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "963bde0f89244407b95cf4653e1016c9" candidate_uuid: "cf9ecdb609f94cf4a215f367199a4903" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227003c258b549cdbfc248d7b4c3b7fa" is_pre_election: true
I20260928 04:47:09.432011 21560 raft_consensus.cc:482] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:47:09.432029 21560 raft_consensus.cc:3037] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:09.432078 21221 raft_consensus.cc:2454] T 963bde0f89244407b95cf4653e1016c9 P 227003c258b549cdbfc248d7b4c3b7fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf9ecdb609f94cf4a215f367199a4903 in term 0.
I20260928 04:47:09.432885 21560 raft_consensus.cc:504] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9ecdb609f94cf4a215f367199a4903" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35799 } } peers { permanent_uuid: "46843b624ba8474e9261e232bd7ab072" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 37133 } } peers { permanent_uuid: "227003c258b549cdbfc248d7b4c3b7fa" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43573 } }
I20260928 04:47:09.433023 21560 leader_election.cc:288] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903 [CANDIDATE]: Term 1 election: Requested vote from peers 46843b624ba8474e9261e232bd7ab072 (127.1.226.195:37133), 227003c258b549cdbfc248d7b4c3b7fa (127.1.226.193:43573)
I20260928 04:47:09.433202 21489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "963bde0f89244407b95cf4653e1016c9" candidate_uuid: "cf9ecdb609f94cf4a215f367199a4903" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46843b624ba8474e9261e232bd7ab072"
I20260928 04:47:09.433207 21221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "963bde0f89244407b95cf4653e1016c9" candidate_uuid: "cf9ecdb609f94cf4a215f367199a4903" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227003c258b549cdbfc248d7b4c3b7fa"
I20260928 04:47:09.433307 21221 raft_consensus.cc:3037] T 963bde0f89244407b95cf4653e1016c9 P 227003c258b549cdbfc248d7b4c3b7fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:09.433307 21489 raft_consensus.cc:3037] T 963bde0f89244407b95cf4653e1016c9 P 46843b624ba8474e9261e232bd7ab072 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:09.434278 21489 raft_consensus.cc:2454] T 963bde0f89244407b95cf4653e1016c9 P 46843b624ba8474e9261e232bd7ab072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf9ecdb609f94cf4a215f367199a4903 in term 1.
I20260928 04:47:09.434278 21221 raft_consensus.cc:2454] T 963bde0f89244407b95cf4653e1016c9 P 227003c258b549cdbfc248d7b4c3b7fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf9ecdb609f94cf4a215f367199a4903 in term 1.
I20260928 04:47:09.434475 21291 leader_election.cc:302] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227003c258b549cdbfc248d7b4c3b7fa, cf9ecdb609f94cf4a215f367199a4903; no voters:
I20260928 04:47:09.434624 21560 raft_consensus.cc:2781] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:09.434847 21560 raft_consensus.cc:686] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903 [term 1 LEADER]: Becoming Leader. State: Replica: cf9ecdb609f94cf4a215f367199a4903, State: Running, Role: LEADER
I20260928 04:47:09.434973 21560 consensus_queue.cc:237] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9ecdb609f94cf4a215f367199a4903" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35799 } } peers { permanent_uuid: "46843b624ba8474e9261e232bd7ab072" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 37133 } } peers { permanent_uuid: "227003c258b549cdbfc248d7b4c3b7fa" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43573 } }
I20260928 04:47:09.435920 21076 catalog_manager.cc:5495] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf9ecdb609f94cf4a215f367199a4903 (127.1.226.194). New cstate: current_term: 1 leader_uuid: "cf9ecdb609f94cf4a215f367199a4903" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9ecdb609f94cf4a215f367199a4903" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 35799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46843b624ba8474e9261e232bd7ab072" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 37133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "227003c258b549cdbfc248d7b4c3b7fa" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43573 } health_report { overall_health: UNKNOWN } } }
W20260928 04:47:09.444130 21268 tablet.cc:2369] T 963bde0f89244407b95cf4653e1016c9 P 227003c258b549cdbfc248d7b4c3b7fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:47:09.469894 21550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:09.467163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57478 (local address 127.1.226.194:35799)
0928 04:47:09.467243 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:09.467247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:09.467260 (+ 13us) server_negotiation.cc:407] Connection header received
0928 04:47:09.467296 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:09.467297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:09.467316 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:09.467380 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:47:09.467588 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.468140 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.469013 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.469222 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.469287 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:09.469494 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:47:09.469602 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:47:09.469672 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:09.469740 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 04:47:09.469766 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 04:47:09.471992 21489 raft_consensus.cc:1261] T 963bde0f89244407b95cf4653e1016c9 P 46843b624ba8474e9261e232bd7ab072 [term 1 FOLLOWER]: Refusing update from remote peer cf9ecdb609f94cf4a215f367199a4903: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:47:09.471992 21221 raft_consensus.cc:1261] T 963bde0f89244407b95cf4653e1016c9 P 227003c258b549cdbfc248d7b4c3b7fa [term 1 FOLLOWER]: Refusing update from remote peer cf9ecdb609f94cf4a215f367199a4903: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:47:09.472461 21562 consensus_queue.cc:1035] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46843b624ba8474e9261e232bd7ab072" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 37133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:47:09.472642 21560 consensus_queue.cc:1035] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227003c258b549cdbfc248d7b4c3b7fa" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 43573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:47:09.477476 21076 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:47:09.477604 21076 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:47:09.478444 21076 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b8e901a6ff24b81a573780a9c96b135: Sending DeleteTablet for 3 replicas of tablet 963bde0f89244407b95cf4653e1016c9
I20260928 04:47:09.478783 21335 tablet_service.cc:1514] Processing DeleteTablet for tablet 963bde0f89244407b95cf4653e1016c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:47:09 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:57474
I20260928 04:47:09.478821 21469 tablet_service.cc:1514] Processing DeleteTablet for tablet 963bde0f89244407b95cf4653e1016c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:47:09 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:45076
I20260928 04:47:09.478852 21201 tablet_service.cc:1514] Processing DeleteTablet for tablet 963bde0f89244407b95cf4653e1016c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:47:09 UTC) from {username='kudu', principal='kudu/127.1.226.254@KRBTEST.COM'} at 127.0.0.1:57172
I20260928 04:47:09.479008 21574 tablet_replica.cc:330] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903: stopping tablet replica
I20260928 04:47:09.479095 21574 raft_consensus.cc:2229] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:47:09.479189 21574 raft_consensus.cc:2258] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:47:09.479315 21576 tablet_replica.cc:330] T 963bde0f89244407b95cf4653e1016c9 P 227003c258b549cdbfc248d7b4c3b7fa: stopping tablet replica
I20260928 04:47:09.479372 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 21138
I20260928 04:47:09.479491 21576 raft_consensus.cc:2229] T 963bde0f89244407b95cf4653e1016c9 P 227003c258b549cdbfc248d7b4c3b7fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:47:09.479614 21576 raft_consensus.cc:2258] T 963bde0f89244407b95cf4653e1016c9 P 227003c258b549cdbfc248d7b4c3b7fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:47:09.479794 21575 tablet_replica.cc:330] T 963bde0f89244407b95cf4653e1016c9 P 46843b624ba8474e9261e232bd7ab072: stopping tablet replica
I20260928 04:47:09.479961 21575 raft_consensus.cc:2229] T 963bde0f89244407b95cf4653e1016c9 P 46843b624ba8474e9261e232bd7ab072 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:47:09.480101 21575 raft_consensus.cc:2258] T 963bde0f89244407b95cf4653e1016c9 P 46843b624ba8474e9261e232bd7ab072 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:47:09.480254 21574 ts_tablet_manager.cc:1905] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:47:09.480656 21575 ts_tablet_manager.cc:1905] T 963bde0f89244407b95cf4653e1016c9 P 46843b624ba8474e9261e232bd7ab072: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:47:09.482146 21574 ts_tablet_manager.cc:1918] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:47:09.482226 21574 log.cc:1196] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-1/wal/wals/963bde0f89244407b95cf4653e1016c9
I20260928 04:47:09.482499 21575 ts_tablet_manager.cc:1918] T 963bde0f89244407b95cf4653e1016c9 P 46843b624ba8474e9261e232bd7ab072: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:47:09.482527 21574 ts_tablet_manager.cc:1939] T 963bde0f89244407b95cf4653e1016c9 P cf9ecdb609f94cf4a215f367199a4903: Deleting consensus metadata
I20260928 04:47:09.482645 21575 log.cc:1196] T 963bde0f89244407b95cf4653e1016c9 P 46843b624ba8474e9261e232bd7ab072: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570585693787-1931-0/minicluster-data/ts-2/wal/wals/963bde0f89244407b95cf4653e1016c9
I20260928 04:47:09.482968 21575 ts_tablet_manager.cc:1939] T 963bde0f89244407b95cf4653e1016c9 P 46843b624ba8474e9261e232bd7ab072: Deleting consensus metadata
I20260928 04:47:09.482986 21061 catalog_manager.cc:4841] TS cf9ecdb609f94cf4a215f367199a4903 (127.1.226.194:35799): tablet 963bde0f89244407b95cf4653e1016c9 (table test-table [id=5f14f99096504dfd8cdcc90e72ac539e]) successfully deleted
I20260928 04:47:09.483435 21063 catalog_manager.cc:4841] TS 46843b624ba8474e9261e232bd7ab072 (127.1.226.195:37133): tablet 963bde0f89244407b95cf4653e1016c9 (table test-table [id=5f14f99096504dfd8cdcc90e72ac539e]) successfully deleted
W20260928 04:47:09.487309 21063 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.226.193:43573 (error 108)
I20260928 04:47:09.487831 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 21272
I20260928 04:47:09.487931 21547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:09.487793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:43573 (local address 127.0.0.1:57180)
0928 04:47:09.487851 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:09.487892 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.193:43573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 04:47:09.489442 21063 catalog_manager.cc:4584] TS 227003c258b549cdbfc248d7b4c3b7fa (127.1.226.193:43573): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 963bde0f89244407b95cf4653e1016c9: Network error: Client connection negotiation failed: client connection to 127.1.226.193:43573: connect: Connection refused (error 111)
I20260928 04:47:09.492965 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 21406
I20260928 04:47:09.497430 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 21044
2026-09-28T04:47:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2058 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T04:47:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:47:09Z Disabled control of system clock
I20260928 04:47:09.525214 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:44391
--webserver_interface=127.1.226.254
--webserver_port=0
--builtin_ntp_servers=127.1.226.212:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:44391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 04:47:09.606207 21581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:09.606426 21581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:09.606447 21581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:09.607518 21581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:09.607553 21581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:09.607565 21581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:09.607577 21581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:09.607589 21581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:09.608947 21581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:44391
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:44391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:09.609143 21581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:09.609380 21581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:09.614406 21586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:09.614493 21581 server_base.cc:1034] running on GCE node
W20260928 04:47:09.614389 21587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:09.614389 21589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:09.615031 21581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:09.615381 21581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:09.616561 21581 hybrid_clock.cc:648] HybridClock initialized: now 1790570829616550 us; error 60 us; skew 500 ppm
I20260928 04:47:09.617799 21581 webserver.cc:460] Webserver started at http://127.1.226.254:43069/ using document root <none> and password file <none>
I20260928 04:47:09.618088 21581 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:09.618160 21581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:09.618310 21581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:09.619377 21581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "cf79e04224514c6ab51f411033bfefed"
format_stamp: "Formatted at 2026-09-28 04:47:09 on dist-test-slave-j5c3"
server_key: "baea532cef8ebe1644a4d413537459e4"
server_key_iv: "c0b77e1232f93e5ebf257c49ab6fec07"
server_key_version: "encryptionkey@0"
I20260928 04:47:09.619737 21581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "cf79e04224514c6ab51f411033bfefed"
format_stamp: "Formatted at 2026-09-28 04:47:09 on dist-test-slave-j5c3"
server_key: "baea532cef8ebe1644a4d413537459e4"
server_key_iv: "c0b77e1232f93e5ebf257c49ab6fec07"
server_key_version: "encryptionkey@0"
I20260928 04:47:09.621137 21581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:47:09.621918 21595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:09.622140 21581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:09.622246 21581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "cf79e04224514c6ab51f411033bfefed"
format_stamp: "Formatted at 2026-09-28 04:47:09 on dist-test-slave-j5c3"
server_key: "baea532cef8ebe1644a4d413537459e4"
server_key_iv: "c0b77e1232f93e5ebf257c49ab6fec07"
server_key_version: "encryptionkey@0"
I20260928 04:47:09.622316 21581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:09.631700 21581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:09.632032 21581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:09.632210 21581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:09.636618 21581 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:44391
I20260928 04:47:09.636660 21647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:44391 every 8 connection(s)
I20260928 04:47:09.637049 21581 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:47:09.638031 21648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:09.639479 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 21581
I20260928 04:47:09.639572 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:47:09.639791 1931 external_mini_cluster.cc:1428] Setting key 90c07906c5a4943c6e8efe39795e73ce
I20260928 04:47:09.640851 21648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf79e04224514c6ab51f411033bfefed: Bootstrap starting.
I20260928 04:47:09.641582 21648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf79e04224514c6ab51f411033bfefed: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:09.641917 21648 log.cc:824] T 00000000000000000000000000000000 P cf79e04224514c6ab51f411033bfefed: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:09.642730 21648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf79e04224514c6ab51f411033bfefed: No bootstrap required, opened a new log
I20260928 04:47:09.644945 21648 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf79e04224514c6ab51f411033bfefed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf79e04224514c6ab51f411033bfefed" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44391 } }
I20260928 04:47:09.645150 21648 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf79e04224514c6ab51f411033bfefed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:09.645191 21648 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf79e04224514c6ab51f411033bfefed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf79e04224514c6ab51f411033bfefed, State: Initialized, Role: FOLLOWER
I20260928 04:47:09.645335 21648 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf79e04224514c6ab51f411033bfefed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf79e04224514c6ab51f411033bfefed" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44391 } }
I20260928 04:47:09.645423 21648 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf79e04224514c6ab51f411033bfefed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:47:09.645459 21648 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf79e04224514c6ab51f411033bfefed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:47:09.645511 21648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf79e04224514c6ab51f411033bfefed [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:09.646138 21648 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf79e04224514c6ab51f411033bfefed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf79e04224514c6ab51f411033bfefed" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44391 } }
I20260928 04:47:09.646301 21648 leader_election.cc:302] T 00000000000000000000000000000000 P cf79e04224514c6ab51f411033bfefed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf79e04224514c6ab51f411033bfefed; no voters:
I20260928 04:47:09.646356 21651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:09.640430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47270 (local address 127.1.226.254:44391)
0928 04:47:09.640645 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:09.640654 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:09.640687 (+ 33us) server_negotiation.cc:407] Connection header received
0928 04:47:09.641302 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:09.641311 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:09.641391 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:09.641551 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:09.641866 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.643179 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.643977 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.644723 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.644854 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:09.645627 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:09.645631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:09.645632 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:09.645814 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:09.645847 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:09.645859 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:09.645932 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:09.646023 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 04:47:09.646108 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":134,"thread_start_us":79,"threads_started":1}
I20260928 04:47:09.646518 21648 leader_election.cc:288] T 00000000000000000000000000000000 P cf79e04224514c6ab51f411033bfefed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:47:09.646550 21653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf79e04224514c6ab51f411033bfefed [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:09.646661 21653 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf79e04224514c6ab51f411033bfefed [term 1 LEADER]: Becoming Leader. State: Replica: cf79e04224514c6ab51f411033bfefed, State: Running, Role: LEADER
I20260928 04:47:09.646752 21653 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf79e04224514c6ab51f411033bfefed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf79e04224514c6ab51f411033bfefed" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44391 } }
I20260928 04:47:09.646811 21648 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf79e04224514c6ab51f411033bfefed [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:09.647071 21653 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf79e04224514c6ab51f411033bfefed [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf79e04224514c6ab51f411033bfefed. Latest consensus state: current_term: 1 leader_uuid: "cf79e04224514c6ab51f411033bfefed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf79e04224514c6ab51f411033bfefed" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44391 } } }
I20260928 04:47:09.647122 21654 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf79e04224514c6ab51f411033bfefed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf79e04224514c6ab51f411033bfefed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf79e04224514c6ab51f411033bfefed" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44391 } } }
I20260928 04:47:09.647142 21653 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf79e04224514c6ab51f411033bfefed [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:09.647469 21654 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf79e04224514c6ab51f411033bfefed [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:09.647591 21660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:47:09.648108 21660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:47:09.649858 21660 catalog_manager.cc:1347] Generated new cluster ID: d5047077a97a473f8aa38f66f609f93f
I20260928 04:47:09.649930 21660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:47:09.663767 21660 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:47:09.664494 21660 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:47:09.668931 21660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf79e04224514c6ab51f411033bfefed: Generated new TSK 0
I20260928 04:47:09.669157 21660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:47:09.675971 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.193:0
--local_ip_for_outbound_sockets=127.1.226.193
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:44391
--builtin_ntp_servers=127.1.226.212:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 04:47:09.759253 21672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:09.759462 21672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:09.759485 21672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:09.760819 21672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:09.760864 21672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:09.760891 21672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.193
I20260928 04:47:09.762248 21672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.226.193
--webserver_port=0
--tserver_master_addrs=127.1.226.254:44391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.193
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:09.762502 21672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:09.762823 21672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:09.767830 21677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:09.767976 21680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:09.768056 21678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:09.768347 21672 server_base.cc:1034] running on GCE node
I20260928 04:47:09.768571 21672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:09.768821 21672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:09.769999 21672 hybrid_clock.cc:648] HybridClock initialized: now 1790570829769981 us; error 33 us; skew 500 ppm
I20260928 04:47:09.771446 21672 webserver.cc:460] Webserver started at http://127.1.226.193:36731/ using document root <none> and password file <none>
I20260928 04:47:09.771811 21672 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:09.771895 21672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:09.772066 21672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:09.773300 21672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-0/data/instance:
uuid: "577da9806bc8407a90ff603634eac593"
format_stamp: "Formatted at 2026-09-28 04:47:09 on dist-test-slave-j5c3"
server_key: "9f6d793ef1dbe2b661a28ed3ca54fc9e"
server_key_iv: "ad6ab304309607f174e608c628f6de5c"
server_key_version: "encryptionkey@0"
I20260928 04:47:09.773690 21672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance:
uuid: "577da9806bc8407a90ff603634eac593"
format_stamp: "Formatted at 2026-09-28 04:47:09 on dist-test-slave-j5c3"
server_key: "9f6d793ef1dbe2b661a28ed3ca54fc9e"
server_key_iv: "ad6ab304309607f174e608c628f6de5c"
server_key_version: "encryptionkey@0"
I20260928 04:47:09.775161 21672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:47:09.775987 21686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:09.776232 21672 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:09.776335 21672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-0/wal
uuid: "577da9806bc8407a90ff603634eac593"
format_stamp: "Formatted at 2026-09-28 04:47:09 on dist-test-slave-j5c3"
server_key: "9f6d793ef1dbe2b661a28ed3ca54fc9e"
server_key_iv: "ad6ab304309607f174e608c628f6de5c"
server_key_version: "encryptionkey@0"
I20260928 04:47:09.776425 21672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:09.805305 21672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:09.805660 21672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:09.805830 21672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:09.806082 21672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:47:09.806437 21672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:47:09.806491 21672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:09.806526 21672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:47:09.806571 21672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:09.812742 21672 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.193:41891
I20260928 04:47:09.812789 21799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.193:41891 every 8 connection(s)
I20260928 04:47:09.813177 21672 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-0/data/info.pb
I20260928 04:47:09.818670 21651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:09.814015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:35249 (local address 127.1.226.254:44391)
0928 04:47:09.814076 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:09.814078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:09.814318 (+ 240us) server_negotiation.cc:407] Connection header received
0928 04:47:09.814825 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:09.814826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:09.814841 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:09.814900 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:09.815549 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.816052 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.817172 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.817368 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.817425 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:09.818351 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:09.818352 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:09.818353 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:09.818357 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:09.818371 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:09.818375 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:09.818429 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:09.818581 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 04:47:09.818606 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 04:47:09.818859 21802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:09.814016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44391 (local address 127.1.226.193:35249)
0928 04:47:09.814186 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:09.814227 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:09.814704 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:09.814954 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:09.814957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:09.814965 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:09.815469 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:09.815473 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.816141 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.816142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:09.817095 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:09.817098 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.817466 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.817467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:09.817594 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:09.818172 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:47:09.818172 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:09.818205 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:47:09.818229 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:09.818462 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:09.818468 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:09.818471 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:47:09.818546 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 04:47:09.818633 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
I20260928 04:47:09.819471 21800 heartbeater.cc:347] Connected to a master server at 127.1.226.254:44391
I20260928 04:47:09.819599 21800 heartbeater.cc:464] Registering TS with master...
I20260928 04:47:09.819799 21800 heartbeater.cc:511] Master 127.1.226.254:44391 requested a full tablet report, sending...
I20260928 04:47:09.820495 21612 ts_manager.cc:194] Registered new tserver with Master: 577da9806bc8407a90ff603634eac593 (127.1.226.193:41891)
I20260928 04:47:09.821264 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 21672
I20260928 04:47:09.821393 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-0/wal/instance
I20260928 04:47:09.821516 21612 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.226.193:35249
I20260928 04:47:09.821615 1931 external_mini_cluster.cc:1428] Setting key b5475314dbf1c89c4b88a4f9e07ed6b4
I20260928 04:47:09.822661 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.194:0
--local_ip_for_outbound_sockets=127.1.226.194
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:44391
--builtin_ntp_servers=127.1.226.212:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 04:47:09.914036 21803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:09.914289 21803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:09.914325 21803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:09.915480 21803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:09.915550 21803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:09.915592 21803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.194
I20260928 04:47:09.916963 21803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.226.194
--webserver_port=0
--tserver_master_addrs=127.1.226.254:44391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.194
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:09.917228 21803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:09.917569 21803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:09.922947 21808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:09.922964 21811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:09.922947 21809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:09.923243 21803 server_base.cc:1034] running on GCE node
I20260928 04:47:09.923506 21803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:09.923787 21803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:09.924971 21803 hybrid_clock.cc:648] HybridClock initialized: now 1790570829924972 us; error 47 us; skew 500 ppm
I20260928 04:47:09.926548 21803 webserver.cc:460] Webserver started at http://127.1.226.194:45921/ using document root <none> and password file <none>
I20260928 04:47:09.926882 21803 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:09.926968 21803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:09.927132 21803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:09.928465 21803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-1/data/instance:
uuid: "34d9605a2d3a41599b4a61103863d958"
format_stamp: "Formatted at 2026-09-28 04:47:09 on dist-test-slave-j5c3"
server_key: "3521ef296e56109b7e390b7475431720"
server_key_iv: "b33e92e3d20cca1cce8ca6f224ee4d9e"
server_key_version: "encryptionkey@0"
I20260928 04:47:09.928882 21803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance:
uuid: "34d9605a2d3a41599b4a61103863d958"
format_stamp: "Formatted at 2026-09-28 04:47:09 on dist-test-slave-j5c3"
server_key: "3521ef296e56109b7e390b7475431720"
server_key_iv: "b33e92e3d20cca1cce8ca6f224ee4d9e"
server_key_version: "encryptionkey@0"
I20260928 04:47:09.930454 21803 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:47:09.931380 21817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:09.931634 21803 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:09.931716 21803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-1/wal
uuid: "34d9605a2d3a41599b4a61103863d958"
format_stamp: "Formatted at 2026-09-28 04:47:09 on dist-test-slave-j5c3"
server_key: "3521ef296e56109b7e390b7475431720"
server_key_iv: "b33e92e3d20cca1cce8ca6f224ee4d9e"
server_key_version: "encryptionkey@0"
I20260928 04:47:09.931859 21803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:09.941183 21803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:09.941488 21803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:09.941604 21803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:09.941812 21803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:47:09.942188 21803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:47:09.942229 21803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:09.942281 21803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:47:09.942310 21803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:09.949386 21803 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.194:43077
I20260928 04:47:09.949436 21930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.194:43077 every 8 connection(s)
I20260928 04:47:09.949836 21803 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-1/data/info.pb
I20260928 04:47:09.955929 21651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:09.950647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.194:37861 (local address 127.1.226.254:44391)
0928 04:47:09.950710 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:09.950712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:09.951055 (+ 343us) server_negotiation.cc:407] Connection header received
0928 04:47:09.951540 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:09.951542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:09.951553 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:09.951611 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:09.952436 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.953068 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.954216 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.954408 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.954486 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:09.955542 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:09.955543 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:09.955544 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:09.955549 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:09.955563 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:09.955567 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:09.955621 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:09.955831 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 04:47:09.955863 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 04:47:09.956081 21933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:09.950647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44391 (local address 127.1.226.194:37861)
0928 04:47:09.950900 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:09.950950 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:09.951444 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:09.951667 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:09.951671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:09.951679 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:09.952298 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:09.952304 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.953229 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.953230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:09.954118 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:09.954120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:09.954511 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:09.954512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:09.954646 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:09.955369 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:47:09.955370 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:09.955405 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:47:09.955432 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:09.955665 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:09.955672 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:09.955675 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:47:09.955750 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 04:47:09.955861 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":93,"threads_started":1}
I20260928 04:47:09.956733 21931 heartbeater.cc:347] Connected to a master server at 127.1.226.254:44391
I20260928 04:47:09.956914 21931 heartbeater.cc:464] Registering TS with master...
I20260928 04:47:09.957127 21931 heartbeater.cc:511] Master 127.1.226.254:44391 requested a full tablet report, sending...
I20260928 04:47:09.957592 21612 ts_manager.cc:194] Registered new tserver with Master: 34d9605a2d3a41599b4a61103863d958 (127.1.226.194:43077)
I20260928 04:47:09.957996 21612 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.226.194:37861
I20260928 04:47:09.959560 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 21803
I20260928 04:47:09.959681 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-1/wal/instance
I20260928 04:47:09.959955 1931 external_mini_cluster.cc:1428] Setting key 1f0bc503447c3ab15413215e5f693d0a
I20260928 04:47:09.961023 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.226.195:0
--local_ip_for_outbound_sockets=127.1.226.195
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:44391
--builtin_ntp_servers=127.1.226.212:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 04:47:10.044646 21934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:10.044867 21934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:10.044889 21934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:10.046096 21934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:10.046139 21934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:10.046164 21934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.226.195
I20260928 04:47:10.047752 21934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.226.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.226.195
--webserver_port=0
--tserver_master_addrs=127.1.226.254:44391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.226.195
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:10.048062 21934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:10.048404 21934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:10.053428 21940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:10.053421 21939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:10.053421 21942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:10.053651 21934 server_base.cc:1034] running on GCE node
I20260928 04:47:10.053814 21934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:10.054030 21934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:10.055174 21934 hybrid_clock.cc:648] HybridClock initialized: now 1790570830055158 us; error 33 us; skew 500 ppm
I20260928 04:47:10.056581 21934 webserver.cc:460] Webserver started at http://127.1.226.195:36229/ using document root <none> and password file <none>
I20260928 04:47:10.056931 21934 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:10.056988 21934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:10.057178 21934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:10.058393 21934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-2/data/instance:
uuid: "7f344012fe204f63a7c806eca6650641"
format_stamp: "Formatted at 2026-09-28 04:47:10 on dist-test-slave-j5c3"
server_key: "8ca77ac5f3c8e2ba0e947fbe633013e7"
server_key_iv: "b87516f1e3c6cf417f1998c50d23f250"
server_key_version: "encryptionkey@0"
I20260928 04:47:10.058794 21934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance:
uuid: "7f344012fe204f63a7c806eca6650641"
format_stamp: "Formatted at 2026-09-28 04:47:10 on dist-test-slave-j5c3"
server_key: "8ca77ac5f3c8e2ba0e947fbe633013e7"
server_key_iv: "b87516f1e3c6cf417f1998c50d23f250"
server_key_version: "encryptionkey@0"
I20260928 04:47:10.060529 21934 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:47:10.061457 21948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:10.061645 21934 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:10.061839 21934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-2/wal
uuid: "7f344012fe204f63a7c806eca6650641"
format_stamp: "Formatted at 2026-09-28 04:47:10 on dist-test-slave-j5c3"
server_key: "8ca77ac5f3c8e2ba0e947fbe633013e7"
server_key_iv: "b87516f1e3c6cf417f1998c50d23f250"
server_key_version: "encryptionkey@0"
I20260928 04:47:10.061932 21934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:10.073767 21934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:10.074128 21934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:10.074313 21934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:10.074580 21934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:47:10.074945 21934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:47:10.075004 21934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:10.075048 21934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:47:10.075081 21934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:10.082050 21934 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.195:38743
I20260928 04:47:10.082094 22061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.195:38743 every 8 connection(s)
I20260928 04:47:10.082469 21934 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-2/data/info.pb
I20260928 04:47:10.086027 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 21934
I20260928 04:47:10.086163 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-2/wal/instance
I20260928 04:47:10.086376 1931 external_mini_cluster.cc:1428] Setting key a68d50efd9e2c89024be5594491a39cd
I20260928 04:47:10.088593 21651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:10.083355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.195:44203 (local address 127.1.226.254:44391)
0928 04:47:10.083420 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:10.083423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:10.083643 (+ 220us) server_negotiation.cc:407] Connection header received
0928 04:47:10.084139 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:10.084141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:10.084165 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:10.084243 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:10.085023 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:10.085552 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:10.086937 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:10.087240 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:10.087354 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:10.088268 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:10.088269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:10.088269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:10.088280 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:10.088295 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:10.088299 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:10.088370 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:10.088483 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 04:47:10.088511 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 04:47:10.088918 22064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:10.083259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:44391 (local address 127.1.226.195:44203)
0928 04:47:10.083467 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:10.083513 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:10.084038 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:10.084328 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:10.084332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:10.084341 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:10.084908 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:10.084913 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:10.085652 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:10.085653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:10.086850 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:10.086853 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:10.087371 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:10.087372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:10.087547 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:10.088066 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:47:10.088066 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:10.088096 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:47:10.088118 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:10.088372 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:10.088379 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:10.088382 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:47:10.088484 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 04:47:10.088591 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":109,"threads_started":1}
I20260928 04:47:10.089427 22062 heartbeater.cc:347] Connected to a master server at 127.1.226.254:44391
I20260928 04:47:10.089550 22062 heartbeater.cc:464] Registering TS with master...
I20260928 04:47:10.089802 22062 heartbeater.cc:511] Master 127.1.226.254:44391 requested a full tablet report, sending...
I20260928 04:47:10.090209 21612 ts_manager.cc:194] Registered new tserver with Master: 7f344012fe204f63a7c806eca6650641 (127.1.226.195:38743)
I20260928 04:47:10.090615 21612 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.226.195:44203
I20260928 04:47:10.098690 1931 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:47:10.103668 21651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:10.100386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47282 (local address 127.1.226.254:44391)
0928 04:47:10.100442 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:10.100444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:10.100558 (+ 114us) server_negotiation.cc:407] Connection header received
0928 04:47:10.100622 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:10.100623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:10.100633 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:10.100673 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:10.100818 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:10.101290 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:10.102326 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:10.102679 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:10.102776 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:10.103414 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:10.103415 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:10.103415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:10.103419 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:10.103433 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:10.103436 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:10.103478 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:10.103515 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 04:47:10.103537 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 04:47:10.105640 21612 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:47:10.107299 21612 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:47:10.113935 22075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:10.110062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35848 (local address 127.1.226.194:43077)
0928 04:47:10.110275 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:10.110280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:10.110290 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:47:10.110326 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:10.110328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:10.110344 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:10.110424 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:10.110608 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:10.111387 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:10.112973 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:10.113516 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:10.113612 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:10.113717 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:10.113801 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 04:47:10.113839 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":100,"threads_started":1}
I20260928 04:47:10.113924 22070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:10.109860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:43077 (local address 127.0.0.1:35848)
0928 04:47:10.110081 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:10.110091 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:10.110140 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:10.110491 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:10.110491 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:10.110493 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:10.110529 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:10.110531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:10.111483 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:10.111484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:10.112863 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:10.112866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:10.113611 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:10.113612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:10.113690 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:10.113703 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 04:47:10.113787 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 04:47:10.113846 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":55,"thread_start_us":74,"threads_started":1}
I20260928 04:47:10.114053 22071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:10.109630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:38743 (local address 127.0.0.1:60334)
0928 04:47:10.109808 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:10.109856 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:10.109925 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:10.110311 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:10.110314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:10.110316 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:10.110403 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:10.110405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:10.111785 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:10.111786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:10.112925 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:10.112927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:10.113803 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:10.113804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:10.113873 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:10.113876 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:47:10.113909 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 04:47:10.113962 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":43,"thread_start_us":63,"threads_started":1}
I20260928 04:47:10.114055 22073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:10.109630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:41891 (local address 127.0.0.1:59772)
0928 04:47:10.109988 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:10.109998 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:10.110093 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:10.110359 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:10.110360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:10.110362 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:10.110416 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:10.110418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:10.111801 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:10.111802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:10.112809 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:10.112812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:10.113844 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:10.113845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:10.113908 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:10.113911 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:47:10.113955 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 04:47:10.113982 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":6,"thread_start_us":113,"threads_started":1}
I20260928 04:47:10.114248 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:10.109723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59772 (local address 127.1.226.193:41891)
0928 04:47:10.109886 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:10.109891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:10.110060 (+ 169us) server_negotiation.cc:407] Connection header received
0928 04:47:10.110190 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:10.110192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:10.110209 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:10.110262 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:10.110517 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:10.111682 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:10.112897 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:10.113643 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:10.113719 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:10.113815 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:10.114094 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 04:47:10.114142 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260928 04:47:10.114547 22074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:10.109921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60334 (local address 127.1.226.195:38743)
0928 04:47:10.110111 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:10.110115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:10.110124 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:47:10.110167 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:10.110169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:10.110186 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:10.110286 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:10.110493 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:10.111691 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:10.113005 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:10.113698 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:10.114282 (+ 584us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:10.114388 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:10.114416 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 04:47:10.114458 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":78,"threads_started":1}
I20260928 04:47:10.115034 21734 tablet_service.cc:1467] Processing CreateTablet for tablet 5026d53ce8e54810b252b34808d93a21 (DEFAULT_TABLE table=test-table [id=d468df70ba1c401abd05726d70f988f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:47:10.115168 21996 tablet_service.cc:1467] Processing CreateTablet for tablet 5026d53ce8e54810b252b34808d93a21 (DEFAULT_TABLE table=test-table [id=d468df70ba1c401abd05726d70f988f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:47:10.115438 21734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5026d53ce8e54810b252b34808d93a21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:10.115351 21865 tablet_service.cc:1467] Processing CreateTablet for tablet 5026d53ce8e54810b252b34808d93a21 (DEFAULT_TABLE table=test-table [id=d468df70ba1c401abd05726d70f988f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:47:10.115549 21996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5026d53ce8e54810b252b34808d93a21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:10.115640 21865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5026d53ce8e54810b252b34808d93a21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:10.118604 22076 tablet_bootstrap.cc:492] T 5026d53ce8e54810b252b34808d93a21 P 7f344012fe204f63a7c806eca6650641: Bootstrap starting.
I20260928 04:47:10.118969 22077 tablet_bootstrap.cc:492] T 5026d53ce8e54810b252b34808d93a21 P 34d9605a2d3a41599b4a61103863d958: Bootstrap starting.
I20260928 04:47:10.119470 22076 tablet_bootstrap.cc:654] T 5026d53ce8e54810b252b34808d93a21 P 7f344012fe204f63a7c806eca6650641: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:10.119632 22077 tablet_bootstrap.cc:654] T 5026d53ce8e54810b252b34808d93a21 P 34d9605a2d3a41599b4a61103863d958: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:10.119817 22076 log.cc:824] T 5026d53ce8e54810b252b34808d93a21 P 7f344012fe204f63a7c806eca6650641: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:10.119956 22077 log.cc:824] T 5026d53ce8e54810b252b34808d93a21 P 34d9605a2d3a41599b4a61103863d958: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:10.120317 22078 tablet_bootstrap.cc:492] T 5026d53ce8e54810b252b34808d93a21 P 577da9806bc8407a90ff603634eac593: Bootstrap starting.
I20260928 04:47:10.120690 22077 tablet_bootstrap.cc:492] T 5026d53ce8e54810b252b34808d93a21 P 34d9605a2d3a41599b4a61103863d958: No bootstrap required, opened a new log
I20260928 04:47:10.120839 22077 ts_tablet_manager.cc:1397] T 5026d53ce8e54810b252b34808d93a21 P 34d9605a2d3a41599b4a61103863d958: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:47:10.120867 22076 tablet_bootstrap.cc:492] T 5026d53ce8e54810b252b34808d93a21 P 7f344012fe204f63a7c806eca6650641: No bootstrap required, opened a new log
I20260928 04:47:10.120975 22078 tablet_bootstrap.cc:654] T 5026d53ce8e54810b252b34808d93a21 P 577da9806bc8407a90ff603634eac593: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:10.120973 22076 ts_tablet_manager.cc:1397] T 5026d53ce8e54810b252b34808d93a21 P 7f344012fe204f63a7c806eca6650641: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:47:10.121316 22078 log.cc:824] T 5026d53ce8e54810b252b34808d93a21 P 577da9806bc8407a90ff603634eac593: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:10.122876 22077 raft_consensus.cc:348] T 5026d53ce8e54810b252b34808d93a21 P 34d9605a2d3a41599b4a61103863d958 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f344012fe204f63a7c806eca6650641" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 38743 } } peers { permanent_uuid: "577da9806bc8407a90ff603634eac593" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41891 } } peers { permanent_uuid: "34d9605a2d3a41599b4a61103863d958" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43077 } }
I20260928 04:47:10.123047 22077 raft_consensus.cc:374] T 5026d53ce8e54810b252b34808d93a21 P 34d9605a2d3a41599b4a61103863d958 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:10.123096 22077 raft_consensus.cc:729] T 5026d53ce8e54810b252b34808d93a21 P 34d9605a2d3a41599b4a61103863d958 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34d9605a2d3a41599b4a61103863d958, State: Initialized, Role: FOLLOWER
I20260928 04:47:10.123222 22077 consensus_queue.cc:260] T 5026d53ce8e54810b252b34808d93a21 P 34d9605a2d3a41599b4a61103863d958 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f344012fe204f63a7c806eca6650641" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 38743 } } peers { permanent_uuid: "577da9806bc8407a90ff603634eac593" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41891 } } peers { permanent_uuid: "34d9605a2d3a41599b4a61103863d958" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43077 } }
I20260928 04:47:10.123270 22076 raft_consensus.cc:348] T 5026d53ce8e54810b252b34808d93a21 P 7f344012fe204f63a7c806eca6650641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f344012fe204f63a7c806eca6650641" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 38743 } } peers { permanent_uuid: "577da9806bc8407a90ff603634eac593" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41891 } } peers { permanent_uuid: "34d9605a2d3a41599b4a61103863d958" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43077 } }
I20260928 04:47:10.123432 22076 raft_consensus.cc:374] T 5026d53ce8e54810b252b34808d93a21 P 7f344012fe204f63a7c806eca6650641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:10.123466 22076 raft_consensus.cc:729] T 5026d53ce8e54810b252b34808d93a21 P 7f344012fe204f63a7c806eca6650641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f344012fe204f63a7c806eca6650641, State: Initialized, Role: FOLLOWER
I20260928 04:47:10.123560 22077 ts_tablet_manager.cc:1428] T 5026d53ce8e54810b252b34808d93a21 P 34d9605a2d3a41599b4a61103863d958: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:47:10.123597 21931 heartbeater.cc:503] Master 127.1.226.254:44391 was elected leader, sending a full tablet report...
I20260928 04:47:10.123584 22076 consensus_queue.cc:260] T 5026d53ce8e54810b252b34808d93a21 P 7f344012fe204f63a7c806eca6650641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f344012fe204f63a7c806eca6650641" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 38743 } } peers { permanent_uuid: "577da9806bc8407a90ff603634eac593" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41891 } } peers { permanent_uuid: "34d9605a2d3a41599b4a61103863d958" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43077 } }
I20260928 04:47:10.123869 22076 ts_tablet_manager.cc:1428] T 5026d53ce8e54810b252b34808d93a21 P 7f344012fe204f63a7c806eca6650641: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:47:10.123905 22062 heartbeater.cc:503] Master 127.1.226.254:44391 was elected leader, sending a full tablet report...
I20260928 04:47:10.124677 22078 tablet_bootstrap.cc:492] T 5026d53ce8e54810b252b34808d93a21 P 577da9806bc8407a90ff603634eac593: No bootstrap required, opened a new log
I20260928 04:47:10.124754 22078 ts_tablet_manager.cc:1397] T 5026d53ce8e54810b252b34808d93a21 P 577da9806bc8407a90ff603634eac593: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260928 04:47:10.126757 22078 raft_consensus.cc:348] T 5026d53ce8e54810b252b34808d93a21 P 577da9806bc8407a90ff603634eac593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f344012fe204f63a7c806eca6650641" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 38743 } } peers { permanent_uuid: "577da9806bc8407a90ff603634eac593" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41891 } } peers { permanent_uuid: "34d9605a2d3a41599b4a61103863d958" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43077 } }
I20260928 04:47:10.126946 22078 raft_consensus.cc:374] T 5026d53ce8e54810b252b34808d93a21 P 577da9806bc8407a90ff603634eac593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:10.126969 22078 raft_consensus.cc:729] T 5026d53ce8e54810b252b34808d93a21 P 577da9806bc8407a90ff603634eac593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 577da9806bc8407a90ff603634eac593, State: Initialized, Role: FOLLOWER
I20260928 04:47:10.127099 22078 consensus_queue.cc:260] T 5026d53ce8e54810b252b34808d93a21 P 577da9806bc8407a90ff603634eac593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f344012fe204f63a7c806eca6650641" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 38743 } } peers { permanent_uuid: "577da9806bc8407a90ff603634eac593" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41891 } } peers { permanent_uuid: "34d9605a2d3a41599b4a61103863d958" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43077 } }
I20260928 04:47:10.127393 22078 ts_tablet_manager.cc:1428] T 5026d53ce8e54810b252b34808d93a21 P 577da9806bc8407a90ff603634eac593: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 04:47:10.127461 21800 heartbeater.cc:503] Master 127.1.226.254:44391 was elected leader, sending a full tablet report...
W20260928 04:47:10.200455 21932 tablet.cc:2369] T 5026d53ce8e54810b252b34808d93a21 P 34d9605a2d3a41599b4a61103863d958: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:47:10.282979 22084 raft_consensus.cc:482] T 5026d53ce8e54810b252b34808d93a21 P 577da9806bc8407a90ff603634eac593 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:47:10.283114 22084 raft_consensus.cc:504] T 5026d53ce8e54810b252b34808d93a21 P 577da9806bc8407a90ff603634eac593 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f344012fe204f63a7c806eca6650641" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 38743 } } peers { permanent_uuid: "577da9806bc8407a90ff603634eac593" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41891 } } peers { permanent_uuid: "34d9605a2d3a41599b4a61103863d958" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43077 } }
I20260928 04:47:10.283627 22084 leader_election.cc:288] T 5026d53ce8e54810b252b34808d93a21 P 577da9806bc8407a90ff603634eac593 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f344012fe204f63a7c806eca6650641 (127.1.226.195:38743), 34d9605a2d3a41599b4a61103863d958 (127.1.226.194:43077)
I20260928 04:47:10.286706 21802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:10.283875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.194:43077 (local address 127.1.226.193:47749)
0928 04:47:10.283974 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:10.283987 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:10.284035 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:10.284234 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:10.284235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:10.284237 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:10.284316 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:10.284318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:10.285031 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:10.285032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:10.285855 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:10.285858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:10.286465 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:10.286466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:10.286537 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:10.286542 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:47:10.286598 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 04:47:10.286642 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 04:47:10.286749 22075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:10.283955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:47749 (local address 127.1.226.194:43077)
0928 04:47:10.284009 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:10.284011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:10.284026 (+ 15us) server_negotiation.cc:407] Connection header received
0928 04:47:10.284123 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:10.284124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:10.284135 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:10.284226 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:10.284394 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:10.284936 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:10.285952 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:10.286376 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:10.286438 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:10.286445 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:10.286632 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 04:47:10.286678 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 04:47:10.286988 22074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:10.284003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.226.193:40187 (local address 127.1.226.195:38743)
0928 04:47:10.284044 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:10.284046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:10.284122 (+ 76us) server_negotiation.cc:407] Connection header received
0928 04:47:10.284245 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:10.284246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:10.284262 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:10.284325 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:47:10.284536 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:10.285052 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:10.286103 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:10.286504 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:10.286580 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:10.286588 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:10.286885 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 04:47:10.286926 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:47:10.287192 22085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:10.283920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.195:38743 (local address 127.1.226.193:40187)
0928 04:47:10.284056 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:10.284066 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:10.284130 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:10.284344 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:10.284345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:10.284347 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:47:10.284454 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:10.284457 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:10.285139 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:10.285140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:10.286009 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:10.286012 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:10.286636 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:10.286638 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:10.286717 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:10.286722 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:47:10.287070 (+ 348us) client_negotiation.cc:240] Negotiation successful
0928 04:47:10.287116 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
I20260928 04:47:10.287154 21885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5026d53ce8e54810b252b34808d93a21" candidate_uuid: "577da9806bc8407a90ff603634eac593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34d9605a2d3a41599b4a61103863d958" is_pre_election: true
I20260928 04:47:10.287340 21885 raft_consensus.cc:2454] T 5026d53ce8e54810b252b34808d93a21 P 34d9605a2d3a41599b4a61103863d958 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 577da9806bc8407a90ff603634eac593 in term 0.
I20260928 04:47:10.287525 22016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5026d53ce8e54810b252b34808d93a21" candidate_uuid: "577da9806bc8407a90ff603634eac593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f344012fe204f63a7c806eca6650641" is_pre_election: true
I20260928 04:47:10.287624 21687 leader_election.cc:302] T 5026d53ce8e54810b252b34808d93a21 P 577da9806bc8407a90ff603634eac593 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34d9605a2d3a41599b4a61103863d958, 577da9806bc8407a90ff603634eac593; no voters:
I20260928 04:47:10.287655 22016 raft_consensus.cc:2454] T 5026d53ce8e54810b252b34808d93a21 P 7f344012fe204f63a7c806eca6650641 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 577da9806bc8407a90ff603634eac593 in term 0.
I20260928 04:47:10.287742 22084 raft_consensus.cc:2781] T 5026d53ce8e54810b252b34808d93a21 P 577da9806bc8407a90ff603634eac593 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:47:10.287824 22084 raft_consensus.cc:482] T 5026d53ce8e54810b252b34808d93a21 P 577da9806bc8407a90ff603634eac593 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:47:10.287860 22084 raft_consensus.cc:3037] T 5026d53ce8e54810b252b34808d93a21 P 577da9806bc8407a90ff603634eac593 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:10.288568 22084 raft_consensus.cc:504] T 5026d53ce8e54810b252b34808d93a21 P 577da9806bc8407a90ff603634eac593 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f344012fe204f63a7c806eca6650641" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 38743 } } peers { permanent_uuid: "577da9806bc8407a90ff603634eac593" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41891 } } peers { permanent_uuid: "34d9605a2d3a41599b4a61103863d958" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43077 } }
I20260928 04:47:10.288826 22084 leader_election.cc:288] T 5026d53ce8e54810b252b34808d93a21 P 577da9806bc8407a90ff603634eac593 [CANDIDATE]: Term 1 election: Requested vote from peers 7f344012fe204f63a7c806eca6650641 (127.1.226.195:38743), 34d9605a2d3a41599b4a61103863d958 (127.1.226.194:43077)
I20260928 04:47:10.288870 22016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5026d53ce8e54810b252b34808d93a21" candidate_uuid: "577da9806bc8407a90ff603634eac593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f344012fe204f63a7c806eca6650641"
I20260928 04:47:10.288887 21885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5026d53ce8e54810b252b34808d93a21" candidate_uuid: "577da9806bc8407a90ff603634eac593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34d9605a2d3a41599b4a61103863d958"
I20260928 04:47:10.288962 22016 raft_consensus.cc:3037] T 5026d53ce8e54810b252b34808d93a21 P 7f344012fe204f63a7c806eca6650641 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:10.288973 21885 raft_consensus.cc:3037] T 5026d53ce8e54810b252b34808d93a21 P 34d9605a2d3a41599b4a61103863d958 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:10.289896 22016 raft_consensus.cc:2454] T 5026d53ce8e54810b252b34808d93a21 P 7f344012fe204f63a7c806eca6650641 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 577da9806bc8407a90ff603634eac593 in term 1.
I20260928 04:47:10.289896 21885 raft_consensus.cc:2454] T 5026d53ce8e54810b252b34808d93a21 P 34d9605a2d3a41599b4a61103863d958 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 577da9806bc8407a90ff603634eac593 in term 1.
I20260928 04:47:10.290068 21687 leader_election.cc:302] T 5026d53ce8e54810b252b34808d93a21 P 577da9806bc8407a90ff603634eac593 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34d9605a2d3a41599b4a61103863d958, 577da9806bc8407a90ff603634eac593; no voters:
I20260928 04:47:10.290190 22084 raft_consensus.cc:2781] T 5026d53ce8e54810b252b34808d93a21 P 577da9806bc8407a90ff603634eac593 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:10.290390 22084 raft_consensus.cc:686] T 5026d53ce8e54810b252b34808d93a21 P 577da9806bc8407a90ff603634eac593 [term 1 LEADER]: Becoming Leader. State: Replica: 577da9806bc8407a90ff603634eac593, State: Running, Role: LEADER
I20260928 04:47:10.290496 22084 consensus_queue.cc:237] T 5026d53ce8e54810b252b34808d93a21 P 577da9806bc8407a90ff603634eac593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f344012fe204f63a7c806eca6650641" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 38743 } } peers { permanent_uuid: "577da9806bc8407a90ff603634eac593" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41891 } } peers { permanent_uuid: "34d9605a2d3a41599b4a61103863d958" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43077 } }
I20260928 04:47:10.291481 21612 catalog_manager.cc:5495] T 5026d53ce8e54810b252b34808d93a21 P 577da9806bc8407a90ff603634eac593 reported cstate change: term changed from 0 to 1, leader changed from <none> to 577da9806bc8407a90ff603634eac593 (127.1.226.193). New cstate: current_term: 1 leader_uuid: "577da9806bc8407a90ff603634eac593" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f344012fe204f63a7c806eca6650641" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 38743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "577da9806bc8407a90ff603634eac593" member_type: VOTER last_known_addr { host: "127.1.226.193" port: 41891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34d9605a2d3a41599b4a61103863d958" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43077 } health_report { overall_health: UNKNOWN } } }
W20260928 04:47:10.313978 21801 tablet.cc:2369] T 5026d53ce8e54810b252b34808d93a21 P 577da9806bc8407a90ff603634eac593: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:47:10.333052 22063 tablet.cc:2369] T 5026d53ce8e54810b252b34808d93a21 P 7f344012fe204f63a7c806eca6650641: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:47:10.341778 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:10.339155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59776 (local address 127.1.226.193:41891)
0928 04:47:10.339230 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:10.339232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:10.339241 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:47:10.339280 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:10.339281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:10.339304 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:10.339489 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:47:10.339635 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:10.340185 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:10.340966 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:10.341158 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:10.341235 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:10.341406 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:47:10.341517 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:47:10.341617 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:10.341640 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 04:47:10.341661 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 04:47:10.343766 22016 raft_consensus.cc:1261] T 5026d53ce8e54810b252b34808d93a21 P 7f344012fe204f63a7c806eca6650641 [term 1 FOLLOWER]: Refusing update from remote peer 577da9806bc8407a90ff603634eac593: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:47:10.343819 21885 raft_consensus.cc:1261] T 5026d53ce8e54810b252b34808d93a21 P 34d9605a2d3a41599b4a61103863d958 [term 1 FOLLOWER]: Refusing update from remote peer 577da9806bc8407a90ff603634eac593: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:47:10.344147 22086 consensus_queue.cc:1035] T 5026d53ce8e54810b252b34808d93a21 P 577da9806bc8407a90ff603634eac593 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f344012fe204f63a7c806eca6650641" member_type: VOTER last_known_addr { host: "127.1.226.195" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:47:10.344367 22084 consensus_queue.cc:1035] T 5026d53ce8e54810b252b34808d93a21 P 577da9806bc8407a90ff603634eac593 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34d9605a2d3a41599b4a61103863d958" member_type: VOTER last_known_addr { host: "127.1.226.194" port: 43077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:47:10.346997 22090 mvcc.cc:205] Tried to move back new op lower bound from 7334178121084768256 to 7334178120870526976. Current Snapshot: MvccSnapshot[applied={T|T < 7334178121084768256}]
I20260928 04:47:10.349633 21612 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:47:10.349812 21612 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:47:10.350679 21612 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf79e04224514c6ab51f411033bfefed: Sending DeleteTablet for 3 replicas of tablet 5026d53ce8e54810b252b34808d93a21
I20260928 04:47:10.351042 21996 tablet_service.cc:1514] Processing DeleteTablet for tablet 5026d53ce8e54810b252b34808d93a21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:47:10 UTC) from {username='slave'} at 127.0.0.1:60334
I20260928 04:47:10.351118 21865 tablet_service.cc:1514] Processing DeleteTablet for tablet 5026d53ce8e54810b252b34808d93a21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:47:10 UTC) from {username='slave'} at 127.0.0.1:35848
I20260928 04:47:10.351125 21734 tablet_service.cc:1514] Processing DeleteTablet for tablet 5026d53ce8e54810b252b34808d93a21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:47:10 UTC) from {username='slave'} at 127.0.0.1:59772
I20260928 04:47:10.351321 22099 tablet_replica.cc:330] T 5026d53ce8e54810b252b34808d93a21 P 7f344012fe204f63a7c806eca6650641: stopping tablet replica
I20260928 04:47:10.351430 22099 raft_consensus.cc:2229] T 5026d53ce8e54810b252b34808d93a21 P 7f344012fe204f63a7c806eca6650641 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:47:10.351434 22100 tablet_replica.cc:330] T 5026d53ce8e54810b252b34808d93a21 P 34d9605a2d3a41599b4a61103863d958: stopping tablet replica
I20260928 04:47:10.351547 22099 raft_consensus.cc:2258] T 5026d53ce8e54810b252b34808d93a21 P 7f344012fe204f63a7c806eca6650641 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:47:10.351564 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 21672
I20260928 04:47:10.351787 22100 raft_consensus.cc:2229] T 5026d53ce8e54810b252b34808d93a21 P 34d9605a2d3a41599b4a61103863d958 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:47:10.351963 22100 raft_consensus.cc:2258] T 5026d53ce8e54810b252b34808d93a21 P 34d9605a2d3a41599b4a61103863d958 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:47:10.352253 22099 ts_tablet_manager.cc:1905] T 5026d53ce8e54810b252b34808d93a21 P 7f344012fe204f63a7c806eca6650641: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:47:10.352453 22100 ts_tablet_manager.cc:1905] T 5026d53ce8e54810b252b34808d93a21 P 34d9605a2d3a41599b4a61103863d958: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:47:10.354245 22100 ts_tablet_manager.cc:1918] T 5026d53ce8e54810b252b34808d93a21 P 34d9605a2d3a41599b4a61103863d958: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:47:10.354260 22099 ts_tablet_manager.cc:1918] T 5026d53ce8e54810b252b34808d93a21 P 7f344012fe204f63a7c806eca6650641: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:47:10.354352 22100 log.cc:1196] T 5026d53ce8e54810b252b34808d93a21 P 34d9605a2d3a41599b4a61103863d958: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-1/wal/wals/5026d53ce8e54810b252b34808d93a21
I20260928 04:47:10.354350 22099 log.cc:1196] T 5026d53ce8e54810b252b34808d93a21 P 7f344012fe204f63a7c806eca6650641: Deleting WAL directory at /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570585693787-1931-0/minicluster-data/ts-2/wal/wals/5026d53ce8e54810b252b34808d93a21
I20260928 04:47:10.354619 22099 ts_tablet_manager.cc:1939] T 5026d53ce8e54810b252b34808d93a21 P 7f344012fe204f63a7c806eca6650641: Deleting consensus metadata
I20260928 04:47:10.354645 22100 ts_tablet_manager.cc:1939] T 5026d53ce8e54810b252b34808d93a21 P 34d9605a2d3a41599b4a61103863d958: Deleting consensus metadata
I20260928 04:47:10.355091 21598 catalog_manager.cc:4841] TS 7f344012fe204f63a7c806eca6650641 (127.1.226.195:38743): tablet 5026d53ce8e54810b252b34808d93a21 (table test-table [id=d468df70ba1c401abd05726d70f988f7]) successfully deleted
I20260928 04:47:10.355168 21596 catalog_manager.cc:4841] TS 34d9605a2d3a41599b4a61103863d958 (127.1.226.194:43077): tablet 5026d53ce8e54810b252b34808d93a21 (table test-table [id=d468df70ba1c401abd05726d70f988f7]) successfully deleted
I20260928 04:47:10.359030 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 21803
W20260928 04:47:10.359045 21596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.226.193:41891 (error 108)
I20260928 04:47:10.360368 22073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:10.359869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.193:41891 (local address 127.0.0.1:59780)
0928 04:47:10.360258 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:10.360316 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.193:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358}
W20260928 04:47:10.360826 21596 catalog_manager.cc:4584] TS 577da9806bc8407a90ff603634eac593 (127.1.226.193:41891): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5026d53ce8e54810b252b34808d93a21: Network error: Client connection negotiation failed: client connection to 127.1.226.193:41891: connect: Connection refused (error 111)
I20260928 04:47:10.364377 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 21934
I20260928 04:47:10.369439 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 21581
2026-09-28T04:47:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (871 ms)
[----------] 6 tests from EncryptionPolicyTest (8600 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:47:10 dist-test-slave-j5c3 krb5kdc[22105](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:47:10 dist-test-slave-j5c3 krb5kdc[22105](info): set up 2 sockets
Sep 28 04:47:10 dist-test-slave-j5c3 krb5kdc[22105](info): commencing operation
krb5kdc: starting...
W20260928 04:47:11.416380 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:47:11 dist-test-slave-j5c3 krb5kdc[22105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:47:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:47:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:47:11.543517 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33565
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:11.628697 22118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:11.628923 22118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:11.628947 22118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:11.630085 22118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:11.630121 22118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:11.630136 22118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:11.630148 22118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:11.630160 22118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:11.631583 22118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33565
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:11.631806 22118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:11.631991 22118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:11.636861 22123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:11.636917 22118 server_base.cc:1034] running on GCE node
W20260928 04:47:11.636879 22126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:11.636894 22124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:11.637439 22118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:11.637712 22118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:11.638969 22118 hybrid_clock.cc:648] HybridClock initialized: now 1790570831638955 us; error 26 us; skew 500 ppm
Sep 28 04:47:11 dist-test-slave-j5c3 krb5kdc[22105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:11.641573 22118 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:47:11.642035 22118 webserver.cc:460] Webserver started at http://127.0.0.1:43203/ using document root <none> and password file <none>
I20260928 04:47:11.642270 22118 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:11.642341 22118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:11.642489 22118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:11.643539 22118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "b35d69cdbfe546dab250b48700ee2f7f"
format_stamp: "Formatted at 2026-09-28 04:47:11 on dist-test-slave-j5c3"
server_key: "33b4e10682c2ddf554fce074c0f240b3"
server_key_iv: "6576f13247fd2499f76b7488665e70e1"
server_key_version: "encryptionkey@0"
I20260928 04:47:11.643908 22118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "b35d69cdbfe546dab250b48700ee2f7f"
format_stamp: "Formatted at 2026-09-28 04:47:11 on dist-test-slave-j5c3"
server_key: "33b4e10682c2ddf554fce074c0f240b3"
server_key_iv: "6576f13247fd2499f76b7488665e70e1"
server_key_version: "encryptionkey@0"
I20260928 04:47:11.645599 22118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:47:11.646463 22133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:11.646654 22118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:47:11.646754 22118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "b35d69cdbfe546dab250b48700ee2f7f"
format_stamp: "Formatted at 2026-09-28 04:47:11 on dist-test-slave-j5c3"
server_key: "33b4e10682c2ddf554fce074c0f240b3"
server_key_iv: "6576f13247fd2499f76b7488665e70e1"
server_key_version: "encryptionkey@0"
I20260928 04:47:11.646878 22118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:11.667636 22118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:11.669940 22118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:11.670110 22118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:11.674161 22118 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33565
I20260928 04:47:11.674225 22185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33565 every 8 connection(s)
I20260928 04:47:11.674594 22118 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:47:11.675729 22186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:11.678309 22186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b35d69cdbfe546dab250b48700ee2f7f: Bootstrap starting.
I20260928 04:47:11.678967 22186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b35d69cdbfe546dab250b48700ee2f7f: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:11.679137 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 22118
I20260928 04:47:11.679279 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:47:11.679329 22186 log.cc:824] T 00000000000000000000000000000000 P b35d69cdbfe546dab250b48700ee2f7f: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:11.679508 1931 external_mini_cluster.cc:1428] Setting key 199ecb2ca8e8f7df7ed6ca5eead86a99
I20260928 04:47:11.680564 22186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b35d69cdbfe546dab250b48700ee2f7f: No bootstrap required, opened a new log
I20260928 04:47:11.683465 22186 raft_consensus.cc:348] T 00000000000000000000000000000000 P b35d69cdbfe546dab250b48700ee2f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b35d69cdbfe546dab250b48700ee2f7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33565 } }
I20260928 04:47:11.683655 22186 raft_consensus.cc:374] T 00000000000000000000000000000000 P b35d69cdbfe546dab250b48700ee2f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:11.683696 22186 raft_consensus.cc:729] T 00000000000000000000000000000000 P b35d69cdbfe546dab250b48700ee2f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35d69cdbfe546dab250b48700ee2f7f, State: Initialized, Role: FOLLOWER
I20260928 04:47:11.683828 22186 consensus_queue.cc:260] T 00000000000000000000000000000000 P b35d69cdbfe546dab250b48700ee2f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b35d69cdbfe546dab250b48700ee2f7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33565 } }
I20260928 04:47:11.683904 22186 raft_consensus.cc:388] T 00000000000000000000000000000000 P b35d69cdbfe546dab250b48700ee2f7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:47:11.683948 22186 raft_consensus.cc:482] T 00000000000000000000000000000000 P b35d69cdbfe546dab250b48700ee2f7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:47:11.683996 22186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b35d69cdbfe546dab250b48700ee2f7f [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:11.684706 22186 raft_consensus.cc:504] T 00000000000000000000000000000000 P b35d69cdbfe546dab250b48700ee2f7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b35d69cdbfe546dab250b48700ee2f7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33565 } }
I20260928 04:47:11.684806 22186 leader_election.cc:302] T 00000000000000000000000000000000 P b35d69cdbfe546dab250b48700ee2f7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b35d69cdbfe546dab250b48700ee2f7f; no voters:
I20260928 04:47:11.685042 22186 leader_election.cc:288] T 00000000000000000000000000000000 P b35d69cdbfe546dab250b48700ee2f7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:47:11.685073 22191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b35d69cdbfe546dab250b48700ee2f7f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:11.685205 22191 raft_consensus.cc:686] T 00000000000000000000000000000000 P b35d69cdbfe546dab250b48700ee2f7f [term 1 LEADER]: Becoming Leader. State: Replica: b35d69cdbfe546dab250b48700ee2f7f, State: Running, Role: LEADER
I20260928 04:47:11.685295 22191 consensus_queue.cc:237] T 00000000000000000000000000000000 P b35d69cdbfe546dab250b48700ee2f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b35d69cdbfe546dab250b48700ee2f7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33565 } }
I20260928 04:47:11.685333 22186 sys_catalog.cc:564] T 00000000000000000000000000000000 P b35d69cdbfe546dab250b48700ee2f7f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:11.685621 22191 sys_catalog.cc:455] T 00000000000000000000000000000000 P b35d69cdbfe546dab250b48700ee2f7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b35d69cdbfe546dab250b48700ee2f7f. Latest consensus state: current_term: 1 leader_uuid: "b35d69cdbfe546dab250b48700ee2f7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b35d69cdbfe546dab250b48700ee2f7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33565 } } }
I20260928 04:47:11.685699 22191 sys_catalog.cc:458] T 00000000000000000000000000000000 P b35d69cdbfe546dab250b48700ee2f7f [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:11.685694 22192 sys_catalog.cc:455] T 00000000000000000000000000000000 P b35d69cdbfe546dab250b48700ee2f7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b35d69cdbfe546dab250b48700ee2f7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b35d69cdbfe546dab250b48700ee2f7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33565 } } }
I20260928 04:47:11.685832 22192 sys_catalog.cc:458] T 00000000000000000000000000000000 P b35d69cdbfe546dab250b48700ee2f7f [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:11.686662 22196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:47:11.687175 22196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 04:47:11 dist-test-slave-j5c3 krb5kdc[22105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 04:47:11.688820 22196 catalog_manager.cc:1347] Generated new cluster ID: bb80e99d4c634bd687dd138273ba7163
I20260928 04:47:11.688876 22196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:47:11.692636 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:11.680288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58485 (local address 127.0.0.1:33565)
0928 04:47:11.680468 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:11.680477 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:11.680515 (+ 38us) server_negotiation.cc:407] Connection header received
0928 04:47:11.681070 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:11.681077 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:11.681126 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:11.681251 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:11.682267 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:11.683135 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:11.684324 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:11.684831 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:11.684948 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:11.688345 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:11.688349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:11.688349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:11.688361 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:11.690496 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:11.690995 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:11.690998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:11.690998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:11.691096 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:11.691333 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:11.691334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:11.691334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:11.691636 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:11.691812 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:11.692307 (+ 495us) server_negotiation.cc:299] Negotiation successful
0928 04:47:11.692407 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":96,"threads_started":1}
I20260928 04:47:11.693638 1931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:47:11.703009 22196 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:47:11.703711 22196 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:47:11.704370 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:11.696463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32771 (local address 127.0.0.1:33565)
0928 04:47:11.696503 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:11.696506 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:11.696606 (+ 100us) server_negotiation.cc:407] Connection header received
0928 04:47:11.696654 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:11.696655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:11.696665 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:11.696734 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:11.697344 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:11.698170 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:11.699197 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:11.699547 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:11.699614 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:11.701077 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:11.701078 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:11.701079 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:11.701085 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:11.702773 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:11.703282 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:11.703284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:11.703284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:11.703345 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:11.703599 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:11.703600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:11.703600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:11.703735 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:11.703854 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:11.704092 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 04:47:11.704185 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:47:11.710042 22196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b35d69cdbfe546dab250b48700ee2f7f: Generated new TSK 0
I20260928 04:47:11.710253 22196 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:47:11.719913 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 22118
W20260928 04:47:11.724021 22103 connection.cc:535] client connection to 127.0.0.1:33565 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33565): Connection reset by peer (error 104)
2026-09-28T04:47:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:47:11 dist-test-slave-j5c3 krb5kdc[22217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:47:11 dist-test-slave-j5c3 krb5kdc[22217](info): set up 2 sockets
Sep 28 04:47:11 dist-test-slave-j5c3 krb5kdc[22217](info): commencing operation
krb5kdc: starting...
W20260928 04:47:12.764416 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:47:12 dist-test-slave-j5c3 krb5kdc[22217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:47:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:47:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:47:12.892573 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40099
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:12.975489 22230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:12.975723 22230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:12.975746 22230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:12.976884 22230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:12.976919 22230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:12.976939 22230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:12.976951 22230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:12.976964 22230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:12.978453 22230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40099
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:12.978649 22230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:12.978829 22230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:12.983667 22236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:12.983705 22235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:12.983667 22238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:12.983981 22230 server_base.cc:1034] running on GCE node
I20260928 04:47:12.984279 22230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:12.984571 22230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:12.985743 22230 hybrid_clock.cc:648] HybridClock initialized: now 1790570832985736 us; error 39 us; skew 500 ppm
Sep 28 04:47:12 dist-test-slave-j5c3 krb5kdc[22217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:12.988456 22230 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:47:12.988911 22230 webserver.cc:460] Webserver started at http://127.0.0.1:45765/ using document root <none> and password file <none>
I20260928 04:47:12.989153 22230 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:12.989220 22230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:12.989374 22230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:12.990490 22230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "2937c883bd724ac78492d100e6d36e06"
format_stamp: "Formatted at 2026-09-28 04:47:12 on dist-test-slave-j5c3"
server_key: "154249a51097ecc5fa5dbce1cf200c82"
server_key_iv: "1c2300f0a4cb2d5a94c378d26685ebb3"
server_key_version: "encryptionkey@0"
I20260928 04:47:12.990855 22230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "2937c883bd724ac78492d100e6d36e06"
format_stamp: "Formatted at 2026-09-28 04:47:12 on dist-test-slave-j5c3"
server_key: "154249a51097ecc5fa5dbce1cf200c82"
server_key_iv: "1c2300f0a4cb2d5a94c378d26685ebb3"
server_key_version: "encryptionkey@0"
I20260928 04:47:12.992364 22230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 04:47:12.993302 22245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:12.993554 22230 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:12.993664 22230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "2937c883bd724ac78492d100e6d36e06"
format_stamp: "Formatted at 2026-09-28 04:47:12 on dist-test-slave-j5c3"
server_key: "154249a51097ecc5fa5dbce1cf200c82"
server_key_iv: "1c2300f0a4cb2d5a94c378d26685ebb3"
server_key_version: "encryptionkey@0"
I20260928 04:47:12.993736 22230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:13.004374 22230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:13.006585 22230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:13.006770 22230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:13.011109 22230 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40099
I20260928 04:47:13.011196 22297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40099 every 8 connection(s)
I20260928 04:47:13.011556 22230 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:47:13.012670 22298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:13.015028 22298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2937c883bd724ac78492d100e6d36e06: Bootstrap starting.
I20260928 04:47:13.015578 22298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2937c883bd724ac78492d100e6d36e06: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:13.015836 22298 log.cc:824] T 00000000000000000000000000000000 P 2937c883bd724ac78492d100e6d36e06: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:13.016544 22298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2937c883bd724ac78492d100e6d36e06: No bootstrap required, opened a new log
I20260928 04:47:13.017752 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 22230
I20260928 04:47:13.017887 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:47:13.018091 1931 external_mini_cluster.cc:1428] Setting key 3f68638f3abdc6efd07796cbe50a26a8
I20260928 04:47:13.018671 22298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2937c883bd724ac78492d100e6d36e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2937c883bd724ac78492d100e6d36e06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40099 } }
I20260928 04:47:13.018872 22298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2937c883bd724ac78492d100e6d36e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:13.018903 22298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2937c883bd724ac78492d100e6d36e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2937c883bd724ac78492d100e6d36e06, State: Initialized, Role: FOLLOWER
I20260928 04:47:13.019065 22298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2937c883bd724ac78492d100e6d36e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2937c883bd724ac78492d100e6d36e06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40099 } }
I20260928 04:47:13.019363 22298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2937c883bd724ac78492d100e6d36e06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:47:13.019454 22298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2937c883bd724ac78492d100e6d36e06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:47:13.019541 22298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2937c883bd724ac78492d100e6d36e06 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:13.020254 22298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2937c883bd724ac78492d100e6d36e06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2937c883bd724ac78492d100e6d36e06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40099 } }
I20260928 04:47:13.020347 22298 leader_election.cc:302] T 00000000000000000000000000000000 P 2937c883bd724ac78492d100e6d36e06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2937c883bd724ac78492d100e6d36e06; no voters:
I20260928 04:47:13.020599 22298 leader_election.cc:288] T 00000000000000000000000000000000 P 2937c883bd724ac78492d100e6d36e06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:47:13.020618 22303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2937c883bd724ac78492d100e6d36e06 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:13.020773 22303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2937c883bd724ac78492d100e6d36e06 [term 1 LEADER]: Becoming Leader. State: Replica: 2937c883bd724ac78492d100e6d36e06, State: Running, Role: LEADER
I20260928 04:47:13.020925 22298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2937c883bd724ac78492d100e6d36e06 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:13.020925 22303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2937c883bd724ac78492d100e6d36e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2937c883bd724ac78492d100e6d36e06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40099 } }
I20260928 04:47:13.021385 22305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2937c883bd724ac78492d100e6d36e06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2937c883bd724ac78492d100e6d36e06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2937c883bd724ac78492d100e6d36e06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40099 } } }
I20260928 04:47:13.021654 22305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2937c883bd724ac78492d100e6d36e06 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:13.021574 22304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2937c883bd724ac78492d100e6d36e06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2937c883bd724ac78492d100e6d36e06. Latest consensus state: current_term: 1 leader_uuid: "2937c883bd724ac78492d100e6d36e06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2937c883bd724ac78492d100e6d36e06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40099 } } }
I20260928 04:47:13.021901 22304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2937c883bd724ac78492d100e6d36e06 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:13.022078 22312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:47:13.022635 22312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:47:13.024338 22312 catalog_manager.cc:1347] Generated new cluster ID: 9a215fc5b1b04389961cba35802ff04d
I20260928 04:47:13.024399 22312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:47:13.024560 22302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:13.018643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58403 (local address 127.0.0.1:40099)
0928 04:47:13.018826 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:13.018834 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:13.018859 (+ 25us) server_negotiation.cc:407] Connection header received
0928 04:47:13.019017 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:13.019025 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:13.019216 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:13.019403 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:13.019639 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:13.021158 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:13.022331 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:13.022948 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:13.023072 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:13.023842 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:13.023846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:13.023846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:13.024019 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:13.024050 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:13.024061 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:13.024162 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:13.024289 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 04:47:13.024321 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260928 04:47:13.028789 1931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:47:13.033134 22312 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:47:13.033645 22312 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:47:13.038658 22302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:13.032490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57989 (local address 127.0.0.1:40099)
0928 04:47:13.032533 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:13.032536 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:13.032676 (+ 140us) server_negotiation.cc:407] Connection header received
0928 04:47:13.032737 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:13.032738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:13.032749 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:13.032799 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:13.033017 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:13.033726 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:13.037215 (+ 3489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:13.037534 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:13.037619 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:13.038391 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:13.038393 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:13.038393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:13.038398 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:13.038414 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:13.038418 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:13.038483 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:13.038560 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 04:47:13.038574 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:47:13.039412 22312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2937c883bd724ac78492d100e6d36e06: Generated new TSK 0
I20260928 04:47:13.039597 22312 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:47:13.045480 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 22230
W20260928 04:47:13.049772 22215 connection.cc:535] client connection to 127.0.0.1:40099 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40099): Connection reset by peer (error 104)
2026-09-28T04:47:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:47:13 dist-test-slave-j5c3 krb5kdc[22329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:47:13 dist-test-slave-j5c3 krb5kdc[22329](info): set up 2 sockets
Sep 28 04:47:13 dist-test-slave-j5c3 krb5kdc[22329](info): commencing operation
krb5kdc: starting...
W20260928 04:47:14.088768 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:47:14 dist-test-slave-j5c3 krb5kdc[22329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:47:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:47:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:47:14.215253 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46797
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:14.299114 22342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:14.299363 22342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:14.299389 22342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:14.300704 22342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:14.300760 22342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:14.300776 22342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:14.300789 22342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:14.300801 22342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:14.302394 22342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46797
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:14.302672 22342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:14.303038 22342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:14.308334 22350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:14.308321 22347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:14.308444 22342 server_base.cc:1034] running on GCE node
W20260928 04:47:14.308321 22348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:14.308813 22342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:14.309123 22342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:14.310365 22342 hybrid_clock.cc:648] HybridClock initialized: now 1790570834310350 us; error 47 us; skew 500 ppm
Sep 28 04:47:14 dist-test-slave-j5c3 krb5kdc[22329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:14.313159 22342 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:47:14.313660 22342 webserver.cc:460] Webserver started at http://127.0.0.1:44783/ using document root <none> and password file <none>
I20260928 04:47:14.313913 22342 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:14.313982 22342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:14.314127 22342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:14.315184 22342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "9378bcb18d964837afe8560596d167db"
format_stamp: "Formatted at 2026-09-28 04:47:14 on dist-test-slave-j5c3"
server_key: "f03bb558f80dde723193e183245373c3"
server_key_iv: "d84b10e65719abe1f213280e369fac1e"
server_key_version: "encryptionkey@0"
I20260928 04:47:14.315549 22342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "9378bcb18d964837afe8560596d167db"
format_stamp: "Formatted at 2026-09-28 04:47:14 on dist-test-slave-j5c3"
server_key: "f03bb558f80dde723193e183245373c3"
server_key_iv: "d84b10e65719abe1f213280e369fac1e"
server_key_version: "encryptionkey@0"
I20260928 04:47:14.316972 22342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:47:14.317679 22357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:14.317884 22342 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:47:14.317989 22342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "9378bcb18d964837afe8560596d167db"
format_stamp: "Formatted at 2026-09-28 04:47:14 on dist-test-slave-j5c3"
server_key: "f03bb558f80dde723193e183245373c3"
server_key_iv: "d84b10e65719abe1f213280e369fac1e"
server_key_version: "encryptionkey@0"
I20260928 04:47:14.318060 22342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:14.321455 22342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:14.323227 22342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:14.323385 22342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:14.327715 22342 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46797
I20260928 04:47:14.327847 22409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46797 every 8 connection(s)
I20260928 04:47:14.328222 22342 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:47:14.329357 22410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:14.330351 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 22342
I20260928 04:47:14.330476 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:47:14.330674 1931 external_mini_cluster.cc:1428] Setting key da119f72d227f4581bb9cba90e7959e9
I20260928 04:47:14.333099 22410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9378bcb18d964837afe8560596d167db: Bootstrap starting.
I20260928 04:47:14.333360 22413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:14.331250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53841 (local address 127.0.0.1:46797)
0928 04:47:14.331448 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:14.331457 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:14.331482 (+ 25us) server_negotiation.cc:407] Connection header received
0928 04:47:14.331896 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:14.331903 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:14.331931 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:14.332052 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:14.332879 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:14.332882 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:14.332883 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:14.332886 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:14.332924 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:14.332935 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:14.332994 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:14.333082 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 04:47:14.333138 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":105,"threads_started":1}
I20260928 04:47:14.333873 22410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9378bcb18d964837afe8560596d167db: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:14.334226 22410 log.cc:824] T 00000000000000000000000000000000 P 9378bcb18d964837afe8560596d167db: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:14.334316 1931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:47:14.334933 22410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9378bcb18d964837afe8560596d167db: No bootstrap required, opened a new log
I20260928 04:47:14.336788 22413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:14.335786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35251 (local address 127.0.0.1:46797)
0928 04:47:14.335825 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:14.335826 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:14.335892 (+ 66us) server_negotiation.cc:407] Connection header received
0928 04:47:14.335956 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:14.335956 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:14.335960 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:14.336000 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:14.336579 (+ 579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:14.336580 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:14.336580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:14.336584 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:14.336596 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:14.336599 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:14.336640 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:14.336700 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 04:47:14.336710 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:47:14.337508 22410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9378bcb18d964837afe8560596d167db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9378bcb18d964837afe8560596d167db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46797 } }
I20260928 04:47:14.337671 22410 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9378bcb18d964837afe8560596d167db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:14.337715 22410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9378bcb18d964837afe8560596d167db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9378bcb18d964837afe8560596d167db, State: Initialized, Role: FOLLOWER
I20260928 04:47:14.337859 22410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9378bcb18d964837afe8560596d167db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9378bcb18d964837afe8560596d167db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46797 } }
I20260928 04:47:14.337960 22410 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9378bcb18d964837afe8560596d167db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:47:14.338002 22410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9378bcb18d964837afe8560596d167db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:47:14.338053 22410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9378bcb18d964837afe8560596d167db [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:14.338629 22410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9378bcb18d964837afe8560596d167db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9378bcb18d964837afe8560596d167db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46797 } }
I20260928 04:47:14.338757 22410 leader_election.cc:302] T 00000000000000000000000000000000 P 9378bcb18d964837afe8560596d167db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9378bcb18d964837afe8560596d167db; no voters:
I20260928 04:47:14.338970 22410 leader_election.cc:288] T 00000000000000000000000000000000 P 9378bcb18d964837afe8560596d167db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:47:14.338997 22420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9378bcb18d964837afe8560596d167db [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:14.339108 22420 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9378bcb18d964837afe8560596d167db [term 1 LEADER]: Becoming Leader. State: Replica: 9378bcb18d964837afe8560596d167db, State: Running, Role: LEADER
I20260928 04:47:14.339191 22420 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9378bcb18d964837afe8560596d167db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9378bcb18d964837afe8560596d167db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46797 } }
I20260928 04:47:14.339233 22410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9378bcb18d964837afe8560596d167db [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:14.339570 22420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9378bcb18d964837afe8560596d167db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9378bcb18d964837afe8560596d167db. Latest consensus state: current_term: 1 leader_uuid: "9378bcb18d964837afe8560596d167db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9378bcb18d964837afe8560596d167db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46797 } } }
I20260928 04:47:14.339586 22421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9378bcb18d964837afe8560596d167db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9378bcb18d964837afe8560596d167db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9378bcb18d964837afe8560596d167db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46797 } } }
I20260928 04:47:14.339731 22420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9378bcb18d964837afe8560596d167db [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:14.339809 22421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9378bcb18d964837afe8560596d167db [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:14.340340 22429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:47:14.340837 22429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:47:14.342463 22429 catalog_manager.cc:1347] Generated new cluster ID: 2cb9ca73693c4cd6accc2d56ba07674f
I20260928 04:47:14.342520 22429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:47:14.348119 22429 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:47:14.348294 22429 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:47:14.358335 22429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9378bcb18d964837afe8560596d167db: Generated new TSK 0
I20260928 04:47:14.358556 22429 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:47:14.369915 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 22342
2026-09-28T04:47:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:47:14 dist-test-slave-j5c3 krb5kdc[22441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:47:14 dist-test-slave-j5c3 krb5kdc[22441](info): set up 2 sockets
Sep 28 04:47:14 dist-test-slave-j5c3 krb5kdc[22441](info): commencing operation
krb5kdc: starting...
W20260928 04:47:15.404352 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:47:15 dist-test-slave-j5c3 krb5kdc[22441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:47:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:47:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:47:15.524968 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45005
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:15.606211 22454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:15.606439 22454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:15.606462 22454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:15.607549 22454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:15.607583 22454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:15.607597 22454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:15.607610 22454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:15.607622 22454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:15.609027 22454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:15.609241 22454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:15.609495 22454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:15.614367 22460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:15.614444 22462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:15.614382 22459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:15.614557 22454 server_base.cc:1034] running on GCE node
I20260928 04:47:15.614843 22454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:15.615180 22454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:15.616346 22454 hybrid_clock.cc:648] HybridClock initialized: now 1790570835616333 us; error 49 us; skew 500 ppm
Sep 28 04:47:15 dist-test-slave-j5c3 krb5kdc[22441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:15.619046 22454 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:47:15.619504 22454 webserver.cc:460] Webserver started at http://127.0.0.1:44481/ using document root <none> and password file <none>
I20260928 04:47:15.619762 22454 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:15.619833 22454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:15.619987 22454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:15.621255 22454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "2d3f3fd82f4445aaad3ecb2fa5e2695a"
format_stamp: "Formatted at 2026-09-28 04:47:15 on dist-test-slave-j5c3"
server_key: "3a5458f23294eaa660361898f8c7bab5"
server_key_iv: "df9acaacb3e30993dbaa3cdb6b12c3d0"
server_key_version: "encryptionkey@0"
I20260928 04:47:15.621625 22454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "2d3f3fd82f4445aaad3ecb2fa5e2695a"
format_stamp: "Formatted at 2026-09-28 04:47:15 on dist-test-slave-j5c3"
server_key: "3a5458f23294eaa660361898f8c7bab5"
server_key_iv: "df9acaacb3e30993dbaa3cdb6b12c3d0"
server_key_version: "encryptionkey@0"
I20260928 04:47:15.623008 22454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:47:15.623765 22469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:15.623946 22454 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:47:15.624048 22454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "2d3f3fd82f4445aaad3ecb2fa5e2695a"
format_stamp: "Formatted at 2026-09-28 04:47:15 on dist-test-slave-j5c3"
server_key: "3a5458f23294eaa660361898f8c7bab5"
server_key_iv: "df9acaacb3e30993dbaa3cdb6b12c3d0"
server_key_version: "encryptionkey@0"
I20260928 04:47:15.624133 22454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:15.640393 22454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:15.642676 22454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:15.642861 22454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:15.646952 22454 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45005
I20260928 04:47:15.646987 22521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45005 every 8 connection(s)
I20260928 04:47:15.647356 22454 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:47:15.648430 22522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:15.649963 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 22454
I20260928 04:47:15.650082 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:47:15.650271 1931 external_mini_cluster.cc:1428] Setting key 107e72d818bec08c4a1c32b2d2ed909f
I20260928 04:47:15.651144 22522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d3f3fd82f4445aaad3ecb2fa5e2695a: Bootstrap starting.
I20260928 04:47:15.651762 22522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d3f3fd82f4445aaad3ecb2fa5e2695a: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:15.652103 22522 log.cc:824] T 00000000000000000000000000000000 P 2d3f3fd82f4445aaad3ecb2fa5e2695a: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:15.652889 22522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d3f3fd82f4445aaad3ecb2fa5e2695a: No bootstrap required, opened a new log
I20260928 04:47:15.654659 22522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d3f3fd82f4445aaad3ecb2fa5e2695a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3f3fd82f4445aaad3ecb2fa5e2695a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45005 } }
I20260928 04:47:15.654819 22522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d3f3fd82f4445aaad3ecb2fa5e2695a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:15.654881 22522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d3f3fd82f4445aaad3ecb2fa5e2695a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d3f3fd82f4445aaad3ecb2fa5e2695a, State: Initialized, Role: FOLLOWER
I20260928 04:47:15.655069 22522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d3f3fd82f4445aaad3ecb2fa5e2695a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3f3fd82f4445aaad3ecb2fa5e2695a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45005 } }
I20260928 04:47:15.655164 22522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d3f3fd82f4445aaad3ecb2fa5e2695a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:47:15.655197 22522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d3f3fd82f4445aaad3ecb2fa5e2695a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:47:15.655277 22522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d3f3fd82f4445aaad3ecb2fa5e2695a [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:15.655946 22522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d3f3fd82f4445aaad3ecb2fa5e2695a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3f3fd82f4445aaad3ecb2fa5e2695a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45005 } }
I20260928 04:47:15.656031 22522 leader_election.cc:302] T 00000000000000000000000000000000 P 2d3f3fd82f4445aaad3ecb2fa5e2695a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d3f3fd82f4445aaad3ecb2fa5e2695a; no voters:
I20260928 04:47:15.656250 22522 leader_election.cc:288] T 00000000000000000000000000000000 P 2d3f3fd82f4445aaad3ecb2fa5e2695a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:47:15.656280 22527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d3f3fd82f4445aaad3ecb2fa5e2695a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:15.656433 22527 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d3f3fd82f4445aaad3ecb2fa5e2695a [term 1 LEADER]: Becoming Leader. State: Replica: 2d3f3fd82f4445aaad3ecb2fa5e2695a, State: Running, Role: LEADER
I20260928 04:47:15.656527 22522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d3f3fd82f4445aaad3ecb2fa5e2695a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:15.656586 22527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d3f3fd82f4445aaad3ecb2fa5e2695a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3f3fd82f4445aaad3ecb2fa5e2695a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45005 } }
I20260928 04:47:15.656951 22527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d3f3fd82f4445aaad3ecb2fa5e2695a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d3f3fd82f4445aaad3ecb2fa5e2695a. Latest consensus state: current_term: 1 leader_uuid: "2d3f3fd82f4445aaad3ecb2fa5e2695a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3f3fd82f4445aaad3ecb2fa5e2695a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45005 } } }
I20260928 04:47:15.657034 22527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d3f3fd82f4445aaad3ecb2fa5e2695a [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:15.656960 22528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d3f3fd82f4445aaad3ecb2fa5e2695a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d3f3fd82f4445aaad3ecb2fa5e2695a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d3f3fd82f4445aaad3ecb2fa5e2695a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45005 } } }
Sep 28 04:47:15 dist-test-slave-j5c3 krb5kdc[22441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 04:47:15.657215 22528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d3f3fd82f4445aaad3ecb2fa5e2695a [sys.catalog]: This master's current role is: LEADER
W20260928 04:47:15.658250 22542 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2d3f3fd82f4445aaad3ecb2fa5e2695a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:47:15.658783 22542 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:47:15.658866 22534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:47:15.659022 22534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:47:15.660597 22534 catalog_manager.cc:1347] Generated new cluster ID: ae47a61eef6c487488e6aa20a08c6fa9
I20260928 04:47:15.660704 22534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:47:15.661162 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:15.650857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55251 (local address 127.0.0.1:45005)
0928 04:47:15.651033 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:15.651041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:15.651066 (+ 25us) server_negotiation.cc:407] Connection header received
0928 04:47:15.651538 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:15.651546 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:15.651596 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:15.651716 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:15.652546 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:15.653506 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:15.654265 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:15.654663 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:15.654793 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:15.657703 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:15.657707 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:15.657708 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:15.657721 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:15.659539 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:15.659924 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:15.659927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:15.659927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:15.660030 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:15.660276 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:15.660277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:15.660277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:15.660578 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:15.660750 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:15.660869 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 04:47:15.661010 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":90,"threads_started":1}
I20260928 04:47:15.662019 1931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:47:15.675675 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:15.664603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42623 (local address 127.0.0.1:45005)
0928 04:47:15.664653 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:15.664655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:15.668388 (+ 3733us) server_negotiation.cc:407] Connection header received
0928 04:47:15.668483 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:15.668484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:15.668496 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:15.668588 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:15.669223 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:15.669925 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:15.670803 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:15.671056 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:15.671160 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:15.672649 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:15.672650 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:15.672651 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:15.672659 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:15.674304 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:15.674703 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:15.674704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:15.674705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:15.674781 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:15.675005 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:15.675006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:15.675007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:15.675132 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:15.675247 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:15.675436 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 04:47:15.675532 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 04:47:15.703925 22534 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:47:15.704591 22534 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:47:15.711764 22534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d3f3fd82f4445aaad3ecb2fa5e2695a: Generated new TSK 0
I20260928 04:47:15.711982 22534 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:47:15.721438 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 22454
2026-09-28T04:47:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:47:15 dist-test-slave-j5c3 krb5kdc[22553](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:47:15 dist-test-slave-j5c3 krb5kdc[22553](info): set up 2 sockets
Sep 28 04:47:15 dist-test-slave-j5c3 krb5kdc[22553](info): commencing operation
krb5kdc: starting...
W20260928 04:47:16.772691 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:47:16 dist-test-slave-j5c3 krb5kdc[22553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:47:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:47:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:47:16.899056 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43125
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:16.982916 22566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:16.983155 22566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:16.983180 22566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:16.984493 22566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:16.984550 22566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:16.984565 22566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:16.984579 22566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:16.984624 22566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:16.986259 22566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:16.986565 22566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:16.986845 22566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:16.992512 22574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:16.992663 22566 server_base.cc:1034] running on GCE node
W20260928 04:47:16.992514 22572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:16.992728 22571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:16.993064 22566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:16.993392 22566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:16.994604 22566 hybrid_clock.cc:648] HybridClock initialized: now 1790570836994591 us; error 49 us; skew 500 ppm
Sep 28 04:47:16 dist-test-slave-j5c3 krb5kdc[22553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:16.997481 22566 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:47:16.997990 22566 webserver.cc:460] Webserver started at http://127.0.0.1:36345/ using document root <none> and password file <none>
I20260928 04:47:16.998199 22566 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:16.998245 22566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:16.998342 22566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:16.999428 22566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "96999882a00a462eba10962ea8e803fa"
format_stamp: "Formatted at 2026-09-28 04:47:16 on dist-test-slave-j5c3"
server_key: "2367c77f981ce269d7bc195049df1609"
server_key_iv: "a17acd16f780c401997711994b18566b"
server_key_version: "encryptionkey@0"
I20260928 04:47:16.999776 22566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "96999882a00a462eba10962ea8e803fa"
format_stamp: "Formatted at 2026-09-28 04:47:16 on dist-test-slave-j5c3"
server_key: "2367c77f981ce269d7bc195049df1609"
server_key_iv: "a17acd16f780c401997711994b18566b"
server_key_version: "encryptionkey@0"
I20260928 04:47:17.001355 22566 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:47:17.002230 22581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:17.002476 22566 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:47:17.002590 22566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "96999882a00a462eba10962ea8e803fa"
format_stamp: "Formatted at 2026-09-28 04:47:16 on dist-test-slave-j5c3"
server_key: "2367c77f981ce269d7bc195049df1609"
server_key_iv: "a17acd16f780c401997711994b18566b"
server_key_version: "encryptionkey@0"
I20260928 04:47:17.002676 22566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:17.022481 22566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:17.024693 22566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:17.024901 22566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:17.029187 22566 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43125
I20260928 04:47:17.029346 22633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43125 every 8 connection(s)
I20260928 04:47:17.029631 22566 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:47:17.030678 22634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:17.033080 22634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96999882a00a462eba10962ea8e803fa: Bootstrap starting.
I20260928 04:47:17.033692 22634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96999882a00a462eba10962ea8e803fa: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:17.033985 22634 log.cc:824] T 00000000000000000000000000000000 P 96999882a00a462eba10962ea8e803fa: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:17.034616 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 22566
I20260928 04:47:17.034744 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:47:17.034799 22634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96999882a00a462eba10962ea8e803fa: No bootstrap required, opened a new log
I20260928 04:47:17.034962 1931 external_mini_cluster.cc:1428] Setting key 094ded55b236c843fd96337a63f53c23
I20260928 04:47:17.036891 22634 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96999882a00a462eba10962ea8e803fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96999882a00a462eba10962ea8e803fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43125 } }
I20260928 04:47:17.037060 22634 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96999882a00a462eba10962ea8e803fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:17.037124 22634 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96999882a00a462eba10962ea8e803fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96999882a00a462eba10962ea8e803fa, State: Initialized, Role: FOLLOWER
I20260928 04:47:17.037271 22634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96999882a00a462eba10962ea8e803fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96999882a00a462eba10962ea8e803fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43125 } }
I20260928 04:47:17.037371 22634 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96999882a00a462eba10962ea8e803fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:47:17.037424 22634 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96999882a00a462eba10962ea8e803fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:47:17.037551 22634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96999882a00a462eba10962ea8e803fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:17.038132 22634 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96999882a00a462eba10962ea8e803fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96999882a00a462eba10962ea8e803fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43125 } }
I20260928 04:47:17.038219 22634 leader_election.cc:302] T 00000000000000000000000000000000 P 96999882a00a462eba10962ea8e803fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96999882a00a462eba10962ea8e803fa; no voters:
I20260928 04:47:17.038415 22634 leader_election.cc:288] T 00000000000000000000000000000000 P 96999882a00a462eba10962ea8e803fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:47:17.038455 22639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96999882a00a462eba10962ea8e803fa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:17.038627 22639 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96999882a00a462eba10962ea8e803fa [term 1 LEADER]: Becoming Leader. State: Replica: 96999882a00a462eba10962ea8e803fa, State: Running, Role: LEADER
I20260928 04:47:17.038686 22634 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96999882a00a462eba10962ea8e803fa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:17.038760 22639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96999882a00a462eba10962ea8e803fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96999882a00a462eba10962ea8e803fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43125 } }
I20260928 04:47:17.039148 22639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96999882a00a462eba10962ea8e803fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96999882a00a462eba10962ea8e803fa. Latest consensus state: current_term: 1 leader_uuid: "96999882a00a462eba10962ea8e803fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96999882a00a462eba10962ea8e803fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43125 } } }
I20260928 04:47:17.039158 22640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96999882a00a462eba10962ea8e803fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96999882a00a462eba10962ea8e803fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96999882a00a462eba10962ea8e803fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43125 } } }
I20260928 04:47:17.039309 22640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96999882a00a462eba10962ea8e803fa [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:17.039520 22639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96999882a00a462eba10962ea8e803fa [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:17.039582 22648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:47:17.040190 22648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:47:17.041821 22648 catalog_manager.cc:1347] Generated new cluster ID: ffa4a00e67354b1686185b7fa866cea6
I20260928 04:47:17.041874 22648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:47:17.046530 22638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:17.035530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57367 (local address 127.0.0.1:43125)
0928 04:47:17.035707 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:17.035715 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:17.035742 (+ 27us) server_negotiation.cc:407] Connection header received
0928 04:47:17.036913 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:17.036921 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:17.036973 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:17.037138 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:17.037378 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:17.038990 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:17.044249 (+ 5259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:17.044919 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:17.045021 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:17.045832 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:17.045836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:17.045837 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:17.046034 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:17.046067 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:17.046079 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:17.046157 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:17.046240 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 04:47:17.046309 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260928 04:47:17.047519 1931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:47:17.051945 22638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:17.049001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37159 (local address 127.0.0.1:43125)
0928 04:47:17.049044 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:17.049046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:17.049156 (+ 110us) server_negotiation.cc:407] Connection header received
0928 04:47:17.049231 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:17.049232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:17.049244 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:17.049300 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:17.049476 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:17.049945 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:17.050670 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:17.050892 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:17.050979 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:17.051675 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:17.051676 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:17.051677 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:17.051681 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:17.051696 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:17.051700 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:17.051761 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:17.051821 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 04:47:17.051844 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:47:17.075668 22648 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:47:17.076393 22648 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:47:17.088732 22648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96999882a00a462eba10962ea8e803fa: Generated new TSK 0
I20260928 04:47:17.089072 22648 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:47:17.094133 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 22566
2026-09-28T04:47:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1372 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:47:17 dist-test-slave-j5c3 krb5kdc[22665](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:47:17 dist-test-slave-j5c3 krb5kdc[22665](info): set up 2 sockets
Sep 28 04:47:17 dist-test-slave-j5c3 krb5kdc[22665](info): commencing operation
krb5kdc: starting...
W20260928 04:47:18.148826 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:47:18 dist-test-slave-j5c3 krb5kdc[22665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:47:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:47:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:47:18.270799 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36203
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:18.351487 22678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:18.351727 22678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:18.351763 22678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:18.352958 22678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:18.353065 22678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:18.353092 22678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:18.353128 22678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:18.353155 22678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:18.354638 22678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:18.354913 22678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:18.355139 22678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:47:18.360253 22678 server_base.cc:1034] running on GCE node
W20260928 04:47:18.360219 22683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:18.360227 22686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:18.360280 22684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:18.360700 22678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:18.360992 22678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:18.362167 22678 hybrid_clock.cc:648] HybridClock initialized: now 1790570838362158 us; error 49 us; skew 500 ppm
Sep 28 04:47:18 dist-test-slave-j5c3 krb5kdc[22665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:18.364744 22678 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:47:18.365232 22678 webserver.cc:460] Webserver started at http://127.0.0.1:44433/ using document root <none> and password file <none>
I20260928 04:47:18.365491 22678 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:18.365537 22678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:18.365700 22678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:18.366792 22678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "241bb5195211482abef71a8ae26e4c81"
format_stamp: "Formatted at 2026-09-28 04:47:18 on dist-test-slave-j5c3"
server_key: "ec66ab7bff3cd471f4c4ae38b584f4e4"
server_key_iv: "0f68170b98e4384c2731d4f47d252295"
server_key_version: "encryptionkey@0"
I20260928 04:47:18.367165 22678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "241bb5195211482abef71a8ae26e4c81"
format_stamp: "Formatted at 2026-09-28 04:47:18 on dist-test-slave-j5c3"
server_key: "ec66ab7bff3cd471f4c4ae38b584f4e4"
server_key_iv: "0f68170b98e4384c2731d4f47d252295"
server_key_version: "encryptionkey@0"
I20260928 04:47:18.368637 22678 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:47:18.369505 22693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:18.369711 22678 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:18.369817 22678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "241bb5195211482abef71a8ae26e4c81"
format_stamp: "Formatted at 2026-09-28 04:47:18 on dist-test-slave-j5c3"
server_key: "ec66ab7bff3cd471f4c4ae38b584f4e4"
server_key_iv: "0f68170b98e4384c2731d4f47d252295"
server_key_version: "encryptionkey@0"
I20260928 04:47:18.369892 22678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:18.373354 22678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:18.375468 22678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:18.375625 22678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:18.379802 22678 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36203
I20260928 04:47:18.379837 22745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36203 every 8 connection(s)
I20260928 04:47:18.380265 22678 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:47:18.381371 22746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:18.384600 22746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 241bb5195211482abef71a8ae26e4c81: Bootstrap starting.
I20260928 04:47:18.385231 22746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 241bb5195211482abef71a8ae26e4c81: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:18.385506 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 22678
I20260928 04:47:18.385658 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:47:18.385644 22746 log.cc:824] T 00000000000000000000000000000000 P 241bb5195211482abef71a8ae26e4c81: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:18.385888 1931 external_mini_cluster.cc:1428] Setting key c64c8151d516fe5bdeee84129faedece
I20260928 04:47:18.386616 22746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 241bb5195211482abef71a8ae26e4c81: No bootstrap required, opened a new log
I20260928 04:47:18.388727 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:18.386444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36447 (local address 127.0.0.1:36203)
0928 04:47:18.386669 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:18.386679 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:18.386717 (+ 38us) server_negotiation.cc:407] Connection header received
0928 04:47:18.387261 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:18.387271 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:18.387309 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:18.387472 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:18.388226 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:18.388230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:18.388231 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:18.388235 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:18.388286 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:18.388302 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:18.388403 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:18.388469 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 04:47:18.388515 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":102,"threads_started":1}
I20260928 04:47:18.389617 22746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 241bb5195211482abef71a8ae26e4c81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241bb5195211482abef71a8ae26e4c81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36203 } }
I20260928 04:47:18.389761 22746 raft_consensus.cc:374] T 00000000000000000000000000000000 P 241bb5195211482abef71a8ae26e4c81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:18.389820 22746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 241bb5195211482abef71a8ae26e4c81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 241bb5195211482abef71a8ae26e4c81, State: Initialized, Role: FOLLOWER
I20260928 04:47:18.389882 1931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:47:18.389946 22746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 241bb5195211482abef71a8ae26e4c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241bb5195211482abef71a8ae26e4c81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36203 } }
I20260928 04:47:18.390069 22746 raft_consensus.cc:388] T 00000000000000000000000000000000 P 241bb5195211482abef71a8ae26e4c81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:47:18.390116 22746 raft_consensus.cc:482] T 00000000000000000000000000000000 P 241bb5195211482abef71a8ae26e4c81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:47:18.390172 22746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 241bb5195211482abef71a8ae26e4c81 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:18.390771 22746 raft_consensus.cc:504] T 00000000000000000000000000000000 P 241bb5195211482abef71a8ae26e4c81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241bb5195211482abef71a8ae26e4c81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36203 } }
I20260928 04:47:18.390902 22746 leader_election.cc:302] T 00000000000000000000000000000000 P 241bb5195211482abef71a8ae26e4c81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 241bb5195211482abef71a8ae26e4c81; no voters:
I20260928 04:47:18.391198 22746 leader_election.cc:288] T 00000000000000000000000000000000 P 241bb5195211482abef71a8ae26e4c81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:47:18.391229 22755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 241bb5195211482abef71a8ae26e4c81 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:18.391419 22755 raft_consensus.cc:686] T 00000000000000000000000000000000 P 241bb5195211482abef71a8ae26e4c81 [term 1 LEADER]: Becoming Leader. State: Replica: 241bb5195211482abef71a8ae26e4c81, State: Running, Role: LEADER
I20260928 04:47:18.391546 22746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 241bb5195211482abef71a8ae26e4c81 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:18.391631 22755 consensus_queue.cc:237] T 00000000000000000000000000000000 P 241bb5195211482abef71a8ae26e4c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241bb5195211482abef71a8ae26e4c81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36203 } }
I20260928 04:47:18.392040 22755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 241bb5195211482abef71a8ae26e4c81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 241bb5195211482abef71a8ae26e4c81. Latest consensus state: current_term: 1 leader_uuid: "241bb5195211482abef71a8ae26e4c81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241bb5195211482abef71a8ae26e4c81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36203 } } }
I20260928 04:47:18.392242 22755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 241bb5195211482abef71a8ae26e4c81 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:18.392056 22756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 241bb5195211482abef71a8ae26e4c81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "241bb5195211482abef71a8ae26e4c81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241bb5195211482abef71a8ae26e4c81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36203 } } }
I20260928 04:47:18.392484 22756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 241bb5195211482abef71a8ae26e4c81 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:18.392628 22765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:47:18.392908 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:18.391464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40297 (local address 127.0.0.1:36203)
0928 04:47:18.391504 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:18.391507 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:18.391665 (+ 158us) server_negotiation.cc:407] Connection header received
0928 04:47:18.391775 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:18.391776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:18.391782 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:18.391811 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:18.392640 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:18.392642 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:18.392642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:18.392648 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:18.392664 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:18.392670 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:18.392744 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:18.392820 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 04:47:18.392838 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:47:18.393474 22765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:47:18.395596 22765 catalog_manager.cc:1347] Generated new cluster ID: 9b84ed8c14994489b85f014e4f829719
I20260928 04:47:18.395675 22765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:47:18.405061 22765 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:47:18.405284 22765 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:47:18.409590 22765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 241bb5195211482abef71a8ae26e4c81: Generated new TSK 0
I20260928 04:47:18.409862 22765 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:47:18.412444 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 22678
2026-09-28T04:47:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:47:18 dist-test-slave-j5c3 krb5kdc[22777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:47:18 dist-test-slave-j5c3 krb5kdc[22777](info): set up 2 sockets
Sep 28 04:47:18 dist-test-slave-j5c3 krb5kdc[22777](info): commencing operation
krb5kdc: starting...
W20260928 04:47:19.468541 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:47:19 dist-test-slave-j5c3 krb5kdc[22777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:47:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:47:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:47:19.596299 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41649
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:19.680667 22790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:19.680894 22790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:19.680919 22790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:19.682133 22790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:19.682183 22790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:19.682204 22790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:19.682216 22790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:19.682228 22790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:19.683751 22790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:19.684017 22790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:19.684311 22790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:19.689671 22795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:19.689623 22798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:19.689602 22790 server_base.cc:1034] running on GCE node
W20260928 04:47:19.689646 22796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:19.690268 22790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:19.690585 22790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:19.691761 22790 hybrid_clock.cc:648] HybridClock initialized: now 1790570839691778 us; error 80 us; skew 500 ppm
Sep 28 04:47:19 dist-test-slave-j5c3 krb5kdc[22777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:19.694916 22790 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:47:19.695623 22790 webserver.cc:460] Webserver started at http://127.0.0.1:33379/ using document root <none> and password file <none>
I20260928 04:47:19.695967 22790 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:19.696051 22790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:19.696223 22790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:19.697388 22790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "293c3cf17a52453997e23a5ed23f67d6"
format_stamp: "Formatted at 2026-09-28 04:47:19 on dist-test-slave-j5c3"
server_key: "e96f247e4d53a58a8c3b93a23b43367f"
server_key_iv: "d8cd91a37e34dd020c57a95592a2ec20"
server_key_version: "encryptionkey@0"
I20260928 04:47:19.697784 22790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "293c3cf17a52453997e23a5ed23f67d6"
format_stamp: "Formatted at 2026-09-28 04:47:19 on dist-test-slave-j5c3"
server_key: "e96f247e4d53a58a8c3b93a23b43367f"
server_key_iv: "d8cd91a37e34dd020c57a95592a2ec20"
server_key_version: "encryptionkey@0"
I20260928 04:47:19.699244 22790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:47:19.700057 22805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:19.700263 22790 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:19.700374 22790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "293c3cf17a52453997e23a5ed23f67d6"
format_stamp: "Formatted at 2026-09-28 04:47:19 on dist-test-slave-j5c3"
server_key: "e96f247e4d53a58a8c3b93a23b43367f"
server_key_iv: "d8cd91a37e34dd020c57a95592a2ec20"
server_key_version: "encryptionkey@0"
I20260928 04:47:19.700469 22790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:19.720471 22790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:19.722800 22790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:19.722990 22790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:19.727356 22790 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41649
I20260928 04:47:19.727422 22857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41649 every 8 connection(s)
I20260928 04:47:19.727826 22790 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:47:19.728976 22858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:19.731555 22858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 293c3cf17a52453997e23a5ed23f67d6: Bootstrap starting.
I20260928 04:47:19.731703 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 22790
I20260928 04:47:19.731848 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:47:19.732072 1931 external_mini_cluster.cc:1428] Setting key c3450e5467798fa0a611b98811691c55
I20260928 04:47:19.732379 22858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 293c3cf17a52453997e23a5ed23f67d6: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:19.732764 22858 log.cc:824] T 00000000000000000000000000000000 P 293c3cf17a52453997e23a5ed23f67d6: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:19.733727 22858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 293c3cf17a52453997e23a5ed23f67d6: No bootstrap required, opened a new log
I20260928 04:47:19.736364 22858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 293c3cf17a52453997e23a5ed23f67d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293c3cf17a52453997e23a5ed23f67d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41649 } }
I20260928 04:47:19.736546 22858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 293c3cf17a52453997e23a5ed23f67d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:19.736574 22858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 293c3cf17a52453997e23a5ed23f67d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 293c3cf17a52453997e23a5ed23f67d6, State: Initialized, Role: FOLLOWER
I20260928 04:47:19.736745 22858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 293c3cf17a52453997e23a5ed23f67d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293c3cf17a52453997e23a5ed23f67d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41649 } }
I20260928 04:47:19.736814 22858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 293c3cf17a52453997e23a5ed23f67d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:47:19.736860 22858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 293c3cf17a52453997e23a5ed23f67d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:47:19.736917 22858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 293c3cf17a52453997e23a5ed23f67d6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:19.737684 22858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 293c3cf17a52453997e23a5ed23f67d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293c3cf17a52453997e23a5ed23f67d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41649 } }
I20260928 04:47:19.737813 22858 leader_election.cc:302] T 00000000000000000000000000000000 P 293c3cf17a52453997e23a5ed23f67d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 293c3cf17a52453997e23a5ed23f67d6; no voters:
I20260928 04:47:19.737998 22858 leader_election.cc:288] T 00000000000000000000000000000000 P 293c3cf17a52453997e23a5ed23f67d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:47:19.738075 22863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 293c3cf17a52453997e23a5ed23f67d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:19.738266 22863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 293c3cf17a52453997e23a5ed23f67d6 [term 1 LEADER]: Becoming Leader. State: Replica: 293c3cf17a52453997e23a5ed23f67d6, State: Running, Role: LEADER
I20260928 04:47:19.738306 22858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 293c3cf17a52453997e23a5ed23f67d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:19.738474 22863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 293c3cf17a52453997e23a5ed23f67d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293c3cf17a52453997e23a5ed23f67d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41649 } }
I20260928 04:47:19.739187 22865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 293c3cf17a52453997e23a5ed23f67d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 293c3cf17a52453997e23a5ed23f67d6. Latest consensus state: current_term: 1 leader_uuid: "293c3cf17a52453997e23a5ed23f67d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293c3cf17a52453997e23a5ed23f67d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41649 } } }
I20260928 04:47:19.739352 22865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 293c3cf17a52453997e23a5ed23f67d6 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:19.739429 22864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 293c3cf17a52453997e23a5ed23f67d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "293c3cf17a52453997e23a5ed23f67d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293c3cf17a52453997e23a5ed23f67d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41649 } } }
I20260928 04:47:19.739590 22864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 293c3cf17a52453997e23a5ed23f67d6 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:19.739658 22870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:47:19.740343 22870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 04:47:19 dist-test-slave-j5c3 krb5kdc[22777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 04:47:19.742096 22870 catalog_manager.cc:1347] Generated new cluster ID: 4c0b4742d53d4576abb305c865e4b147
I20260928 04:47:19.742156 22870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:47:19.745206 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:19.732721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.8:42291 (local address 127.0.0.1:41649)
0928 04:47:19.732931 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:19.732941 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:19.732981 (+ 40us) server_negotiation.cc:407] Connection header received
0928 04:47:19.733611 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:19.733621 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:19.733693 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:19.733885 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:19.734575 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:19.736277 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:19.737320 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:19.738055 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:19.738576 (+ 521us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:19.741637 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:19.741641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:19.741642 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:19.741656 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:19.743406 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:19.743829 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:19.743833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:19.743833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:19.743955 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:19.744279 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:19.744280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:19.744281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:19.744536 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:19.744718 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:19.744856 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 04:47:19.744965 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":85,"threads_started":1}
I20260928 04:47:19.746143 1931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:47:19.758777 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:19.747538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.8:57175 (local address 127.0.0.1:41649)
0928 04:47:19.747577 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:19.747579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:19.752437 (+ 4858us) server_negotiation.cc:407] Connection header received
0928 04:47:19.752521 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:19.752522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:19.752535 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:19.752585 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:19.753165 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:19.753700 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:19.754453 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:19.754680 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:19.754758 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:19.756009 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:19.756011 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:19.756011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:19.756019 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:19.757547 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:19.757925 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:19.757926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:19.757926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:19.757992 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:19.758215 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:19.758216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:19.758216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:19.758340 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:19.758456 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:19.758587 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 04:47:19.758653 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:47:19.763830 22870 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:47:19.764653 22870 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:47:19.774343 22870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 293c3cf17a52453997e23a5ed23f67d6: Generated new TSK 0
I20260928 04:47:19.774629 22870 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:47:19.785740 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 22790
W20260928 04:47:19.790220 22775 connection.cc:535] client connection to 127.0.0.1:41649 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41649): Connection reset by peer (error 104)
2026-09-28T04:47:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1372 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:47:19 dist-test-slave-j5c3 krb5kdc[22889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:47:19 dist-test-slave-j5c3 krb5kdc[22889](info): set up 2 sockets
Sep 28 04:47:19 dist-test-slave-j5c3 krb5kdc[22889](info): commencing operation
krb5kdc: starting...
W20260928 04:47:20.820348 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:47:20 dist-test-slave-j5c3 krb5kdc[22889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:47:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:47:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:47:20.951990 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44563
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:21.045612 22902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:21.045872 22902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:21.045902 22902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:21.047230 22902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:21.047287 22902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:21.047307 22902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:21.047322 22902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:21.047338 22902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:21.049005 22902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:21.049242 22902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:21.049530 22902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:21.055451 22907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:21.055459 22902 server_base.cc:1034] running on GCE node
W20260928 04:47:21.055420 22908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:21.055420 22910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:21.056028 22902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:21.056383 22902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:21.057562 22902 hybrid_clock.cc:648] HybridClock initialized: now 1790570841057548 us; error 54 us; skew 500 ppm
Sep 28 04:47:21 dist-test-slave-j5c3 krb5kdc[22889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:21.060271 22902 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:47:21.060745 22902 webserver.cc:460] Webserver started at http://127.0.0.1:33257/ using document root <none> and password file <none>
I20260928 04:47:21.060959 22902 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:21.061002 22902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:21.061090 22902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:21.062116 22902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "5008f53bab6846b588927b80a4642e46"
format_stamp: "Formatted at 2026-09-28 04:47:21 on dist-test-slave-j5c3"
server_key: "0bf729ccc9ffb044c225064b1420b8b8"
server_key_iv: "e340680f8531720797b2fe746ea1282b"
server_key_version: "encryptionkey@0"
I20260928 04:47:21.062425 22902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "5008f53bab6846b588927b80a4642e46"
format_stamp: "Formatted at 2026-09-28 04:47:21 on dist-test-slave-j5c3"
server_key: "0bf729ccc9ffb044c225064b1420b8b8"
server_key_iv: "e340680f8531720797b2fe746ea1282b"
server_key_version: "encryptionkey@0"
I20260928 04:47:21.063732 22902 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:47:21.064558 22917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:21.064786 22902 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:21.064882 22902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "5008f53bab6846b588927b80a4642e46"
format_stamp: "Formatted at 2026-09-28 04:47:21 on dist-test-slave-j5c3"
server_key: "0bf729ccc9ffb044c225064b1420b8b8"
server_key_iv: "e340680f8531720797b2fe746ea1282b"
server_key_version: "encryptionkey@0"
I20260928 04:47:21.064970 22902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:21.078678 22902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:21.081202 22902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:21.081415 22902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:21.085481 22902 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44563
I20260928 04:47:21.085577 22969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44563 every 8 connection(s)
I20260928 04:47:21.085939 22902 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:47:21.086989 22970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:21.087800 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 22902
I20260928 04:47:21.087931 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:47:21.088172 1931 external_mini_cluster.cc:1428] Setting key 21dd03e6e3d59a6ee80f2c613e0a9292
I20260928 04:47:21.090229 22970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5008f53bab6846b588927b80a4642e46: Bootstrap starting.
I20260928 04:47:21.090881 22970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5008f53bab6846b588927b80a4642e46: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:21.091192 22970 log.cc:824] T 00000000000000000000000000000000 P 5008f53bab6846b588927b80a4642e46: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:21.091943 22970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5008f53bab6846b588927b80a4642e46: No bootstrap required, opened a new log
I20260928 04:47:21.093914 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:21.088725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.8:35303 (local address 127.0.0.1:44563)
0928 04:47:21.088905 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:21.088914 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:21.088939 (+ 25us) server_negotiation.cc:407] Connection header received
0928 04:47:21.089374 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:21.089381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:21.089446 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:21.089566 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:21.089801 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:21.090753 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:21.091783 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:21.092292 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:21.092380 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:21.093094 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:21.093098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:21.093098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:21.093286 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:21.093333 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:21.093345 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:21.093420 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:21.093640 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 04:47:21.093675 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260928 04:47:21.094336 22970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5008f53bab6846b588927b80a4642e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008f53bab6846b588927b80a4642e46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44563 } }
I20260928 04:47:21.094491 22970 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5008f53bab6846b588927b80a4642e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:21.094527 22970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5008f53bab6846b588927b80a4642e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5008f53bab6846b588927b80a4642e46, State: Initialized, Role: FOLLOWER
I20260928 04:47:21.094688 22970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5008f53bab6846b588927b80a4642e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008f53bab6846b588927b80a4642e46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44563 } }
I20260928 04:47:21.094810 22970 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5008f53bab6846b588927b80a4642e46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:47:21.094856 22970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5008f53bab6846b588927b80a4642e46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:47:21.094930 22970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5008f53bab6846b588927b80a4642e46 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:21.094954 1931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:47:21.095491 22970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5008f53bab6846b588927b80a4642e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008f53bab6846b588927b80a4642e46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44563 } }
I20260928 04:47:21.095575 22970 leader_election.cc:302] T 00000000000000000000000000000000 P 5008f53bab6846b588927b80a4642e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5008f53bab6846b588927b80a4642e46; no voters:
I20260928 04:47:21.095777 22970 leader_election.cc:288] T 00000000000000000000000000000000 P 5008f53bab6846b588927b80a4642e46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:47:21.095804 22975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5008f53bab6846b588927b80a4642e46 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:21.095929 22975 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5008f53bab6846b588927b80a4642e46 [term 1 LEADER]: Becoming Leader. State: Replica: 5008f53bab6846b588927b80a4642e46, State: Running, Role: LEADER
I20260928 04:47:21.096056 22970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5008f53bab6846b588927b80a4642e46 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:21.096064 22975 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5008f53bab6846b588927b80a4642e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008f53bab6846b588927b80a4642e46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44563 } }
I20260928 04:47:21.096540 22978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5008f53bab6846b588927b80a4642e46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5008f53bab6846b588927b80a4642e46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008f53bab6846b588927b80a4642e46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44563 } } }
I20260928 04:47:21.096556 22977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5008f53bab6846b588927b80a4642e46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5008f53bab6846b588927b80a4642e46. Latest consensus state: current_term: 1 leader_uuid: "5008f53bab6846b588927b80a4642e46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5008f53bab6846b588927b80a4642e46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44563 } } }
I20260928 04:47:21.096653 22978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5008f53bab6846b588927b80a4642e46 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:21.096665 22977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5008f53bab6846b588927b80a4642e46 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:21.096889 22987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:47:21.097461 22987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:47:21.099025 22987 catalog_manager.cc:1347] Generated new cluster ID: 05558b4dce764ed79ca0870a20315d94
I20260928 04:47:21.099076 22987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:47:21.100726 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:21.096357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.8:34571 (local address 127.0.0.1:44563)
0928 04:47:21.096389 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:21.096391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:21.096986 (+ 595us) server_negotiation.cc:407] Connection header received
0928 04:47:21.097079 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:21.097080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:21.097090 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:21.097133 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:21.097388 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:21.098158 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:21.099237 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:21.099541 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:21.099612 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:21.100408 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:21.100409 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:21.100409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:21.100414 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:21.100429 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:21.100433 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:21.100483 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:21.100594 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 04:47:21.100604 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 04:47:21.116345 22987 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:47:21.117106 22987 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:47:21.121485 22987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5008f53bab6846b588927b80a4642e46: Generated new TSK 0
I20260928 04:47:21.121723 22987 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:47:21.124712 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 22902
W20260928 04:47:21.129307 22887 connection.cc:535] client connection to 127.0.0.1:44563 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44563): Connection reset by peer (error 104)
2026-09-28T04:47:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:47:21 dist-test-slave-j5c3 krb5kdc[23001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:47:21 dist-test-slave-j5c3 krb5kdc[23001](info): set up 2 sockets
Sep 28 04:47:21 dist-test-slave-j5c3 krb5kdc[23001](info): commencing operation
krb5kdc: starting...
W20260928 04:47:22.264336 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:47:22 dist-test-slave-j5c3 krb5kdc[23001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:47:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:47:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:47:22.390723 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46443
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:22.474967 23014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:22.475243 23014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:22.475291 23014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:22.476521 23014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:22.476563 23014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:22.476578 23014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:22.476590 23014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:22.476632 23014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:22.478153 23014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:22.478412 23014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:22.478657 23014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:22.483706 23022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:22.483745 23019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:22.483717 23020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:22.483994 23014 server_base.cc:1034] running on GCE node
I20260928 04:47:22.484292 23014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:22.484594 23014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:22.485750 23014 hybrid_clock.cc:648] HybridClock initialized: now 1790570842485736 us; error 34 us; skew 500 ppm
Sep 28 04:47:22 dist-test-slave-j5c3 krb5kdc[23001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:22.488539 23014 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:47:22.489094 23014 webserver.cc:460] Webserver started at http://127.0.0.1:33765/ using document root <none> and password file <none>
I20260928 04:47:22.489390 23014 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:22.489487 23014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:22.489660 23014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:22.490872 23014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "6a594dc1a7d04a74956321621e8d9aeb"
format_stamp: "Formatted at 2026-09-28 04:47:22 on dist-test-slave-j5c3"
server_key: "7693e84918dacabea3f4be9c0f2ee022"
server_key_iv: "649e3735171bcce637c90d861a7305ee"
server_key_version: "encryptionkey@0"
I20260928 04:47:22.491293 23014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "6a594dc1a7d04a74956321621e8d9aeb"
format_stamp: "Formatted at 2026-09-28 04:47:22 on dist-test-slave-j5c3"
server_key: "7693e84918dacabea3f4be9c0f2ee022"
server_key_iv: "649e3735171bcce637c90d861a7305ee"
server_key_version: "encryptionkey@0"
I20260928 04:47:22.493002 23014 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260928 04:47:22.493898 23029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:22.494165 23014 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:22.494297 23014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "6a594dc1a7d04a74956321621e8d9aeb"
format_stamp: "Formatted at 2026-09-28 04:47:22 on dist-test-slave-j5c3"
server_key: "7693e84918dacabea3f4be9c0f2ee022"
server_key_iv: "649e3735171bcce637c90d861a7305ee"
server_key_version: "encryptionkey@0"
I20260928 04:47:22.494386 23014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:22.498059 23014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:22.500625 23014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:22.500870 23014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:22.504925 23014 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46443
I20260928 04:47:22.504954 23081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46443 every 8 connection(s)
I20260928 04:47:22.505291 23014 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:47:22.505896 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 23014
I20260928 04:47:22.506055 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:47:22.506285 1931 external_mini_cluster.cc:1428] Setting key 5cb9c26332f0e09489de94b62504ca08
I20260928 04:47:22.506572 23082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:22.509121 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:22.506934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.8:50413 (local address 127.0.0.1:46443)
0928 04:47:22.507142 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:22.507151 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:22.507175 (+ 24us) server_negotiation.cc:407] Connection header received
0928 04:47:22.507688 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:22.507695 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:22.507721 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:22.507921 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:22.508597 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:22.508601 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:22.508601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:22.508605 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:22.508653 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:22.508663 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:22.508846 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:22.508918 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 04:47:22.508962 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":91,"threads_started":1}
I20260928 04:47:22.509927 23082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a594dc1a7d04a74956321621e8d9aeb: Bootstrap starting.
I20260928 04:47:22.510185 1931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:47:22.510584 23082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a594dc1a7d04a74956321621e8d9aeb: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:22.510879 23082 log.cc:824] T 00000000000000000000000000000000 P 6a594dc1a7d04a74956321621e8d9aeb: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:22.511672 23082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a594dc1a7d04a74956321621e8d9aeb: No bootstrap required, opened a new log
I20260928 04:47:22.512858 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:22.511618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.8:39753 (local address 127.0.0.1:46443)
0928 04:47:22.511653 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:22.511655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:22.511791 (+ 136us) server_negotiation.cc:407] Connection header received
0928 04:47:22.511892 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:22.511893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:22.511898 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:22.511938 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:22.512582 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:22.512584 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:22.512584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:22.512587 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:22.512601 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:22.512604 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:22.512657 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:22.512745 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 04:47:22.512759 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 04:47:22.514106 23082 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a594dc1a7d04a74956321621e8d9aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a594dc1a7d04a74956321621e8d9aeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46443 } }
I20260928 04:47:22.514251 23082 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a594dc1a7d04a74956321621e8d9aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:22.514276 23082 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a594dc1a7d04a74956321621e8d9aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a594dc1a7d04a74956321621e8d9aeb, State: Initialized, Role: FOLLOWER
I20260928 04:47:22.514402 23082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a594dc1a7d04a74956321621e8d9aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a594dc1a7d04a74956321621e8d9aeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46443 } }
I20260928 04:47:22.514489 23082 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a594dc1a7d04a74956321621e8d9aeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:47:22.514532 23082 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a594dc1a7d04a74956321621e8d9aeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:47:22.514618 23082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a594dc1a7d04a74956321621e8d9aeb [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:22.515172 23082 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a594dc1a7d04a74956321621e8d9aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a594dc1a7d04a74956321621e8d9aeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46443 } }
I20260928 04:47:22.515298 23082 leader_election.cc:302] T 00000000000000000000000000000000 P 6a594dc1a7d04a74956321621e8d9aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a594dc1a7d04a74956321621e8d9aeb; no voters:
I20260928 04:47:22.515486 23082 leader_election.cc:288] T 00000000000000000000000000000000 P 6a594dc1a7d04a74956321621e8d9aeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:47:22.515528 23092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a594dc1a7d04a74956321621e8d9aeb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:22.515724 23092 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a594dc1a7d04a74956321621e8d9aeb [term 1 LEADER]: Becoming Leader. State: Replica: 6a594dc1a7d04a74956321621e8d9aeb, State: Running, Role: LEADER
I20260928 04:47:22.515733 23082 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a594dc1a7d04a74956321621e8d9aeb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:22.515825 23092 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a594dc1a7d04a74956321621e8d9aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a594dc1a7d04a74956321621e8d9aeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46443 } }
I20260928 04:47:22.516268 23094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a594dc1a7d04a74956321621e8d9aeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a594dc1a7d04a74956321621e8d9aeb. Latest consensus state: current_term: 1 leader_uuid: "6a594dc1a7d04a74956321621e8d9aeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a594dc1a7d04a74956321621e8d9aeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46443 } } }
I20260928 04:47:22.516255 23093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a594dc1a7d04a74956321621e8d9aeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a594dc1a7d04a74956321621e8d9aeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a594dc1a7d04a74956321621e8d9aeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46443 } } }
I20260928 04:47:22.516415 23094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a594dc1a7d04a74956321621e8d9aeb [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:22.516425 23093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a594dc1a7d04a74956321621e8d9aeb [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:22.516764 23101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:47:22.517307 23101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:47:22.519061 23101 catalog_manager.cc:1347] Generated new cluster ID: 43fd3b250ebb49b5a69b3f78f2dfab25
I20260928 04:47:22.519111 23101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:47:22.543316 23101 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:47:22.543522 23101 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:47:22.551028 23101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a594dc1a7d04a74956321621e8d9aeb: Generated new TSK 0
I20260928 04:47:22.551268 23101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:47:22.559598 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 23014
2026-09-28T04:47:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1434 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:47:22 dist-test-slave-j5c3 krb5kdc[23113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:47:22 dist-test-slave-j5c3 krb5kdc[23113](info): set up 2 sockets
Sep 28 04:47:22 dist-test-slave-j5c3 krb5kdc[23113](info): commencing operation
krb5kdc: starting...
W20260928 04:47:23.600452 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:47:23 dist-test-slave-j5c3 krb5kdc[23113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:47:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:47:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:47:23.723367 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38283
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:23.805753 23126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:23.805966 23126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:23.805989 23126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:23.807078 23126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:23.807116 23126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:23.807129 23126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:23.807142 23126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:23.807153 23126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:23.808600 23126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38283
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:23.808799 23126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:23.808972 23126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:23.813898 23134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:23.813903 23132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:23.813903 23131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:23.814139 23126 server_base.cc:1034] running on GCE node
I20260928 04:47:23.814486 23126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:23.814805 23126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:23.815989 23126 hybrid_clock.cc:648] HybridClock initialized: now 1790570843815967 us; error 42 us; skew 500 ppm
Sep 28 04:47:23 dist-test-slave-j5c3 krb5kdc[23113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:23.818686 23126 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:47:23.819172 23126 webserver.cc:460] Webserver started at http://127.0.0.1:44841/ using document root <none> and password file <none>
I20260928 04:47:23.819383 23126 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:23.819424 23126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:23.819528 23126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:23.820690 23126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "1c301e53322b4a0bb780ed238e1fcb34"
format_stamp: "Formatted at 2026-09-28 04:47:23 on dist-test-slave-j5c3"
server_key: "ec633d11681dc8ef26b1c0aa3181ccab"
server_key_iv: "eb89d66d205a27c7289ff2e2d666274f"
server_key_version: "encryptionkey@0"
I20260928 04:47:23.821015 23126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "1c301e53322b4a0bb780ed238e1fcb34"
format_stamp: "Formatted at 2026-09-28 04:47:23 on dist-test-slave-j5c3"
server_key: "ec633d11681dc8ef26b1c0aa3181ccab"
server_key_iv: "eb89d66d205a27c7289ff2e2d666274f"
server_key_version: "encryptionkey@0"
I20260928 04:47:23.822324 23126 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:47:23.823029 23141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:23.823237 23126 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:23.823329 23126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "1c301e53322b4a0bb780ed238e1fcb34"
format_stamp: "Formatted at 2026-09-28 04:47:23 on dist-test-slave-j5c3"
server_key: "ec633d11681dc8ef26b1c0aa3181ccab"
server_key_iv: "eb89d66d205a27c7289ff2e2d666274f"
server_key_version: "encryptionkey@0"
I20260928 04:47:23.823386 23126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:23.839700 23126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:23.842161 23126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:23.842283 23126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:23.846105 23126 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38283
I20260928 04:47:23.846179 23193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38283 every 8 connection(s)
I20260928 04:47:23.846540 23126 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:47:23.847584 23194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:23.848412 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 23126
I20260928 04:47:23.848536 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:47:23.848713 1931 external_mini_cluster.cc:1428] Setting key c649173b4237e2c50c9bea801babe681
I20260928 04:47:23.850943 23194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c301e53322b4a0bb780ed238e1fcb34: Bootstrap starting.
I20260928 04:47:23.851537 23194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c301e53322b4a0bb780ed238e1fcb34: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:23.851836 23194 log.cc:824] T 00000000000000000000000000000000 P 1c301e53322b4a0bb780ed238e1fcb34: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:23.852772 23194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c301e53322b4a0bb780ed238e1fcb34: No bootstrap required, opened a new log
I20260928 04:47:23.854872 23194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c301e53322b4a0bb780ed238e1fcb34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c301e53322b4a0bb780ed238e1fcb34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38283 } }
I20260928 04:47:23.855010 23194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c301e53322b4a0bb780ed238e1fcb34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:23.855032 23194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c301e53322b4a0bb780ed238e1fcb34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c301e53322b4a0bb780ed238e1fcb34, State: Initialized, Role: FOLLOWER
I20260928 04:47:23.855177 23194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c301e53322b4a0bb780ed238e1fcb34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c301e53322b4a0bb780ed238e1fcb34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38283 } }
I20260928 04:47:23.855278 23194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c301e53322b4a0bb780ed238e1fcb34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:47:23.855324 23194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c301e53322b4a0bb780ed238e1fcb34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:47:23.855399 23194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c301e53322b4a0bb780ed238e1fcb34 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 04:47:23 dist-test-slave-j5c3 krb5kdc[23113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 04:47:23.856050 23194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c301e53322b4a0bb780ed238e1fcb34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c301e53322b4a0bb780ed238e1fcb34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38283 } }
I20260928 04:47:23.856189 23194 leader_election.cc:302] T 00000000000000000000000000000000 P 1c301e53322b4a0bb780ed238e1fcb34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c301e53322b4a0bb780ed238e1fcb34; no voters:
I20260928 04:47:23.856400 23194 leader_election.cc:288] T 00000000000000000000000000000000 P 1c301e53322b4a0bb780ed238e1fcb34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:47:23.856457 23199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c301e53322b4a0bb780ed238e1fcb34 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:23.856616 23199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c301e53322b4a0bb780ed238e1fcb34 [term 1 LEADER]: Becoming Leader. State: Replica: 1c301e53322b4a0bb780ed238e1fcb34, State: Running, Role: LEADER
I20260928 04:47:23.856690 23194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c301e53322b4a0bb780ed238e1fcb34 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:23.856779 23199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c301e53322b4a0bb780ed238e1fcb34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c301e53322b4a0bb780ed238e1fcb34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38283 } }
I20260928 04:47:23.857600 23200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c301e53322b4a0bb780ed238e1fcb34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c301e53322b4a0bb780ed238e1fcb34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c301e53322b4a0bb780ed238e1fcb34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38283 } } }
I20260928 04:47:23.857774 23200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c301e53322b4a0bb780ed238e1fcb34 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:23.858029 23209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:47:23.858554 23209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:47:23.858678 23201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c301e53322b4a0bb780ed238e1fcb34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c301e53322b4a0bb780ed238e1fcb34. Latest consensus state: current_term: 1 leader_uuid: "1c301e53322b4a0bb780ed238e1fcb34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c301e53322b4a0bb780ed238e1fcb34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38283 } } }
I20260928 04:47:23.858908 23201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c301e53322b4a0bb780ed238e1fcb34 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:23.860258 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:23.849235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.8:57475 (local address 127.0.0.1:38283)
0928 04:47:23.849445 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:23.849453 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:23.849479 (+ 26us) server_negotiation.cc:407] Connection header received
0928 04:47:23.849903 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:23.849918 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:23.849969 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:23.850084 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:23.850957 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:23.852026 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:23.853227 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:23.853647 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:23.853787 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:23.856354 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:23.856360 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:23.856360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:23.856378 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:23.858542 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:23.858988 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:23.858991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:23.858991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:23.859103 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:23.859391 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:23.859392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:23.859393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:23.859697 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:23.859858 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:23.859939 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 04:47:23.860040 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":118,"threads_started":1}
I20260928 04:47:23.860882 23209 catalog_manager.cc:1347] Generated new cluster ID: 04ce678cbf9748e4bfc12be27df3fdaf
I20260928 04:47:23.860934 23209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:47:23.861238 1931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:47:23.873018 23209 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:47:23.873449 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:23.866369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.8:46543 (local address 127.0.0.1:38283)
0928 04:47:23.866427 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:23.866430 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:23.866620 (+ 190us) server_negotiation.cc:407] Connection header received
0928 04:47:23.866703 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:23.866704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:23.866716 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:23.866783 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:23.867384 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:23.867954 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:23.868793 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:23.869038 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:23.869130 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:23.870441 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:23.870443 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:23.870443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:23.870451 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:23.871918 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:23.872396 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:23.872398 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:23.872398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:23.872460 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:23.872744 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:23.872745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:23.872746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:23.872887 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:23.873032 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:23.873275 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 04:47:23.873343 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 04:47:23.873739 23209 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:47:23.884331 23209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c301e53322b4a0bb780ed238e1fcb34: Generated new TSK 0
I20260928 04:47:23.884542 23209 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:47:23.891510 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 23126
W20260928 04:47:23.895443 23111 connection.cc:535] client connection to 127.0.0.1:38283 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38283): Connection reset by peer (error 104)
2026-09-28T04:47:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:47:23 dist-test-slave-j5c3 krb5kdc[23225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:47:23 dist-test-slave-j5c3 krb5kdc[23225](info): set up 2 sockets
Sep 28 04:47:23 dist-test-slave-j5c3 krb5kdc[23225](info): commencing operation
krb5kdc: starting...
W20260928 04:47:24.968492 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:47:25 dist-test-slave-j5c3 krb5kdc[23225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:47:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:47:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:47:25.090855 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33829
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:25.172739 23238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:25.172951 23238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:25.172977 23238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:25.174165 23238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:25.174212 23238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:25.174227 23238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:25.174276 23238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:25.174302 23238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:25.176133 23238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33829
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:25.176434 23238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:25.176697 23238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:47:25.181917 23238 server_base.cc:1034] running on GCE node
W20260928 04:47:25.181895 23244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:25.181851 23246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:25.181885 23243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:25.182371 23238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:25.182703 23238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:25.183847 23238 hybrid_clock.cc:648] HybridClock initialized: now 1790570845183839 us; error 33 us; skew 500 ppm
Sep 28 04:47:25 dist-test-slave-j5c3 krb5kdc[23225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:25.186637 23238 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:47:25.187175 23238 webserver.cc:460] Webserver started at http://127.0.0.1:40501/ using document root <none> and password file <none>
I20260928 04:47:25.187433 23238 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:25.187502 23238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:25.187655 23238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:25.188815 23238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "bdf3bdce29e64624929a1f0baadded88"
format_stamp: "Formatted at 2026-09-28 04:47:25 on dist-test-slave-j5c3"
server_key: "a85f27527376391e4ed537f4c5fa77fb"
server_key_iv: "f3dbe0fd9a5b94bc5c36c62fe072b571"
server_key_version: "encryptionkey@0"
I20260928 04:47:25.189204 23238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "bdf3bdce29e64624929a1f0baadded88"
format_stamp: "Formatted at 2026-09-28 04:47:25 on dist-test-slave-j5c3"
server_key: "a85f27527376391e4ed537f4c5fa77fb"
server_key_iv: "f3dbe0fd9a5b94bc5c36c62fe072b571"
server_key_version: "encryptionkey@0"
I20260928 04:47:25.190591 23238 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:47:25.191476 23253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:25.191697 23238 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:25.191830 23238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "bdf3bdce29e64624929a1f0baadded88"
format_stamp: "Formatted at 2026-09-28 04:47:25 on dist-test-slave-j5c3"
server_key: "a85f27527376391e4ed537f4c5fa77fb"
server_key_iv: "f3dbe0fd9a5b94bc5c36c62fe072b571"
server_key_version: "encryptionkey@0"
I20260928 04:47:25.191912 23238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:25.203799 23238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:25.206315 23238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:25.206496 23238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:25.210790 23238 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33829
I20260928 04:47:25.210829 23305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33829 every 8 connection(s)
I20260928 04:47:25.211205 23238 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:47:25.212100 23306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:25.214519 23306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdf3bdce29e64624929a1f0baadded88: Bootstrap starting.
I20260928 04:47:25.215073 23306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdf3bdce29e64624929a1f0baadded88: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:25.215303 23306 log.cc:824] T 00000000000000000000000000000000 P bdf3bdce29e64624929a1f0baadded88: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:25.215992 23306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdf3bdce29e64624929a1f0baadded88: No bootstrap required, opened a new log
I20260928 04:47:25.216055 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 23238
I20260928 04:47:25.216224 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:47:25.216428 1931 external_mini_cluster.cc:1428] Setting key 82750d78595c133464ff1ddeefd05dd1
I20260928 04:47:25.218410 23306 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdf3bdce29e64624929a1f0baadded88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf3bdce29e64624929a1f0baadded88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33829 } }
I20260928 04:47:25.218595 23306 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdf3bdce29e64624929a1f0baadded88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:25.218638 23306 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdf3bdce29e64624929a1f0baadded88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf3bdce29e64624929a1f0baadded88, State: Initialized, Role: FOLLOWER
I20260928 04:47:25.218788 23306 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdf3bdce29e64624929a1f0baadded88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf3bdce29e64624929a1f0baadded88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33829 } }
I20260928 04:47:25.218863 23306 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdf3bdce29e64624929a1f0baadded88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:47:25.218894 23306 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdf3bdce29e64624929a1f0baadded88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:47:25.218940 23306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdf3bdce29e64624929a1f0baadded88 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:25.219636 23306 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdf3bdce29e64624929a1f0baadded88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf3bdce29e64624929a1f0baadded88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33829 } }
I20260928 04:47:25.219779 23306 leader_election.cc:302] T 00000000000000000000000000000000 P bdf3bdce29e64624929a1f0baadded88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdf3bdce29e64624929a1f0baadded88; no voters:
I20260928 04:47:25.219939 23306 leader_election.cc:288] T 00000000000000000000000000000000 P bdf3bdce29e64624929a1f0baadded88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:47:25.220136 23312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdf3bdce29e64624929a1f0baadded88 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:25.220304 23306 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdf3bdce29e64624929a1f0baadded88 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:25.220319 23312 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdf3bdce29e64624929a1f0baadded88 [term 1 LEADER]: Becoming Leader. State: Replica: bdf3bdce29e64624929a1f0baadded88, State: Running, Role: LEADER
I20260928 04:47:25.220556 23312 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdf3bdce29e64624929a1f0baadded88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf3bdce29e64624929a1f0baadded88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33829 } }
I20260928 04:47:25.220963 23312 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdf3bdce29e64624929a1f0baadded88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdf3bdce29e64624929a1f0baadded88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf3bdce29e64624929a1f0baadded88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33829 } } }
I20260928 04:47:25.221091 23312 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdf3bdce29e64624929a1f0baadded88 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:25.221268 23312 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdf3bdce29e64624929a1f0baadded88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdf3bdce29e64624929a1f0baadded88. Latest consensus state: current_term: 1 leader_uuid: "bdf3bdce29e64624929a1f0baadded88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf3bdce29e64624929a1f0baadded88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33829 } } }
I20260928 04:47:25.221441 23312 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdf3bdce29e64624929a1f0baadded88 [sys.catalog]: This master's current role is: LEADER
W20260928 04:47:25.222350 23326 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bdf3bdce29e64624929a1f0baadded88: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:47:25.222471 23326 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:47:25.223017 23320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:47:25.223493 23320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:47:25.223958 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:25.216970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.8:52847 (local address 127.0.0.1:33829)
0928 04:47:25.217176 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:25.217187 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:25.217221 (+ 34us) server_negotiation.cc:407] Connection header received
0928 04:47:25.217968 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:25.217978 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:25.218048 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:25.218217 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:25.218381 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:25.220059 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:25.221277 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:25.222065 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:25.222171 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:25.223135 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:25.223138 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:25.223139 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:25.223336 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:25.223374 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:25.223388 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:25.223461 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:25.223693 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 04:47:25.223721 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":87,"threads_started":1}
I20260928 04:47:25.225277 1931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:47:25.227581 23320 catalog_manager.cc:1347] Generated new cluster ID: ed18c0e0ffe24804808b7fb2ae3514be
I20260928 04:47:25.227629 23320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:47:25.231361 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:25.226831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.8:34453 (local address 127.0.0.1:33829)
0928 04:47:25.226945 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:25.226947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:25.227059 (+ 112us) server_negotiation.cc:407] Connection header received
0928 04:47:25.227159 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:25.227160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:25.227172 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:25.227263 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:25.228220 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:25.228913 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:25.230016 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:25.230324 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:25.230385 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:25.231103 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:25.231105 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:25.231105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:25.231109 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:25.231123 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:25.231127 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:25.231180 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:25.231280 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 04:47:25.231290 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 04:47:25.237807 23320 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:47:25.238551 23320 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:47:25.247233 23320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdf3bdce29e64624929a1f0baadded88: Generated new TSK 0
I20260928 04:47:25.247457 23320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:47:25.255360 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 23238
W20260928 04:47:25.259533 23223 connection.cc:535] client connection to 127.0.0.1:33829 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33829): Connection reset by peer (error 104)
2026-09-28T04:47:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1363 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:47:25 dist-test-slave-j5c3 krb5kdc[23337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:47:25 dist-test-slave-j5c3 krb5kdc[23337](info): set up 2 sockets
Sep 28 04:47:25 dist-test-slave-j5c3 krb5kdc[23337](info): commencing operation
krb5kdc: starting...
W20260928 04:47:26.340474 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:47:26 dist-test-slave-j5c3 krb5kdc[23337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:47:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:47:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:47:26.462390 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33669
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:26.543712 23350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:26.543941 23350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:26.543964 23350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:26.545215 23350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:26.545269 23350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:26.545284 23350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:26.545295 23350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:26.545307 23350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:26.546900 23350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33669
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570585693787-1931-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:26.547164 23350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:26.547418 23350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:26.552848 23356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:26.552830 23355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:26.552917 23358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:26.553083 23350 server_base.cc:1034] running on GCE node
I20260928 04:47:26.553454 23350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:26.553771 23350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:26.554981 23350 hybrid_clock.cc:648] HybridClock initialized: now 1790570846554946 us; error 49 us; skew 500 ppm
Sep 28 04:47:26 dist-test-slave-j5c3 krb5kdc[23337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:26.557686 23350 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:47:26.558139 23350 webserver.cc:460] Webserver started at http://127.0.0.1:43527/ using document root <none> and password file <none>
I20260928 04:47:26.558398 23350 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:26.558475 23350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:26.558627 23350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:26.559707 23350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "cd551936032e452190db1ca24febde55"
format_stamp: "Formatted at 2026-09-28 04:47:26 on dist-test-slave-j5c3"
server_key: "cacdc64aff085667a148c40f13895d90"
server_key_iv: "ad32e17a26abde5777efc9c9266b6231"
server_key_version: "encryptionkey@0"
I20260928 04:47:26.560081 23350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "cd551936032e452190db1ca24febde55"
format_stamp: "Formatted at 2026-09-28 04:47:26 on dist-test-slave-j5c3"
server_key: "cacdc64aff085667a148c40f13895d90"
server_key_iv: "ad32e17a26abde5777efc9c9266b6231"
server_key_version: "encryptionkey@0"
I20260928 04:47:26.561458 23350 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:47:26.562223 23365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:26.562430 23350 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:47:26.562538 23350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "cd551936032e452190db1ca24febde55"
format_stamp: "Formatted at 2026-09-28 04:47:26 on dist-test-slave-j5c3"
server_key: "cacdc64aff085667a148c40f13895d90"
server_key_iv: "ad32e17a26abde5777efc9c9266b6231"
server_key_version: "encryptionkey@0"
I20260928 04:47:26.562613 23350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:26.565886 23350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:26.567742 23350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:26.567895 23350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:26.571661 23350 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33669
I20260928 04:47:26.571692 23417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33669 every 8 connection(s)
I20260928 04:47:26.572046 23350 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:47:26.573119 23418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:26.575601 23418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd551936032e452190db1ca24febde55: Bootstrap starting.
I20260928 04:47:26.576300 23418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd551936032e452190db1ca24febde55: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:26.576613 23418 log.cc:824] T 00000000000000000000000000000000 P cd551936032e452190db1ca24febde55: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:26.577495 23418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd551936032e452190db1ca24febde55: No bootstrap required, opened a new log
I20260928 04:47:26.577479 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 23350
I20260928 04:47:26.577646 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:47:26.577950 1931 external_mini_cluster.cc:1428] Setting key e0e7ec60d5227c4d8b62ee2539a377ba
I20260928 04:47:26.579620 23418 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd551936032e452190db1ca24febde55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd551936032e452190db1ca24febde55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33669 } }
I20260928 04:47:26.579803 23418 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd551936032e452190db1ca24febde55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:26.579854 23418 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd551936032e452190db1ca24febde55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd551936032e452190db1ca24febde55, State: Initialized, Role: FOLLOWER
I20260928 04:47:26.580041 23418 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd551936032e452190db1ca24febde55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd551936032e452190db1ca24febde55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33669 } }
I20260928 04:47:26.580201 23418 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd551936032e452190db1ca24febde55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:47:26.580252 23418 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd551936032e452190db1ca24febde55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:47:26.580307 23418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd551936032e452190db1ca24febde55 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:26.581175 23422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:26.578492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.8:37059 (local address 127.0.0.1:33669)
0928 04:47:26.578689 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:26.578699 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:26.578737 (+ 38us) server_negotiation.cc:407] Connection header received
0928 04:47:26.579601 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:26.579609 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:26.579638 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:26.579788 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:26.580686 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:26.580691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:26.580691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:26.580696 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:26.580748 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:26.580761 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:26.580851 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:26.580932 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 04:47:26.580981 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
I20260928 04:47:26.581396 23418 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd551936032e452190db1ca24febde55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd551936032e452190db1ca24febde55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33669 } }
I20260928 04:47:26.581514 23418 leader_election.cc:302] T 00000000000000000000000000000000 P cd551936032e452190db1ca24febde55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd551936032e452190db1ca24febde55; no voters:
I20260928 04:47:26.581722 23418 leader_election.cc:288] T 00000000000000000000000000000000 P cd551936032e452190db1ca24febde55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:47:26.581760 23423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd551936032e452190db1ca24febde55 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:26.581880 23423 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd551936032e452190db1ca24febde55 [term 1 LEADER]: Becoming Leader. State: Replica: cd551936032e452190db1ca24febde55, State: Running, Role: LEADER
I20260928 04:47:26.581969 23423 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd551936032e452190db1ca24febde55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd551936032e452190db1ca24febde55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33669 } }
I20260928 04:47:26.582021 23418 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd551936032e452190db1ca24febde55 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:26.582273 1931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:47:26.582413 23424 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd551936032e452190db1ca24febde55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd551936032e452190db1ca24febde55. Latest consensus state: current_term: 1 leader_uuid: "cd551936032e452190db1ca24febde55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd551936032e452190db1ca24febde55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33669 } } }
I20260928 04:47:26.582569 23424 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd551936032e452190db1ca24febde55 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:26.582412 23425 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd551936032e452190db1ca24febde55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd551936032e452190db1ca24febde55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd551936032e452190db1ca24febde55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33669 } } }
I20260928 04:47:26.582769 23425 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd551936032e452190db1ca24febde55 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:26.582893 23431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:47:26.583539 23431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:47:26.585490 23422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:26.584071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.8:38217 (local address 127.0.0.1:33669)
0928 04:47:26.584114 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:26.584116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:26.584294 (+ 178us) server_negotiation.cc:407] Connection header received
0928 04:47:26.584401 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:26.584402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:26.584409 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:26.584467 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:26.585219 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:26.585221 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:26.585221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:26.585226 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:26.585243 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:26.585249 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:26.585330 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:26.585404 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 04:47:26.585421 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:47:26.585696 23431 catalog_manager.cc:1347] Generated new cluster ID: b02a362f725943d09968d9791b57a4d9
I20260928 04:47:26.585760 23431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:47:26.621372 23431 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:47:26.621618 23431 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:47:26.628340 23431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd551936032e452190db1ca24febde55: Generated new TSK 0
I20260928 04:47:26.628585 23431 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:47:26.634895 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 23350
2026-09-28T04:47:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:47:26 dist-test-slave-j5c3 krb5kdc[23449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:47:26 dist-test-slave-j5c3 krb5kdc[23449](info): set up 2 sockets
Sep 28 04:47:26 dist-test-slave-j5c3 krb5kdc[23449](info): commencing operation
krb5kdc: starting...
W20260928 04:47:27.700399 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:47:27 dist-test-slave-j5c3 krb5kdc[23449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:47:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:47:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:47:27.827015 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:40465
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:37729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:40465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:27.909556 23462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:27.909835 23462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:27.909878 23462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:27.911087 23462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:27.911164 23462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:27.911187 23462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:27.911221 23462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:27.911252 23462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:27.912833 23462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:37729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:40465
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:40465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:27.913125 23462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:27.913365 23462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:27.918320 23470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:27.918315 23468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:27.918313 23467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:27.918638 23462 server_base.cc:1034] running on GCE node
I20260928 04:47:27.918989 23462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:27.919310 23462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:27.920502 23462 hybrid_clock.cc:648] HybridClock initialized: now 1790570847920487 us; error 35 us; skew 500 ppm
Sep 28 04:47:27 dist-test-slave-j5c3 krb5kdc[23449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570847, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:27.927577 23462 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:47:27.928144 23462 webserver.cc:460] Webserver started at http://127.1.226.254:38271/ using document root <none> and password file <none>
I20260928 04:47:27.928468 23462 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:27.928525 23462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:27.928913 23462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:27.930248 23462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "a8a0bba4a36c4aacb1a9fac99c938485"
format_stamp: "Formatted at 2026-09-28 04:47:27 on dist-test-slave-j5c3"
server_key: "f08f9db25070c1338c7f84d764556615"
server_key_iv: "c2d1d090bad356574b372c4486e7072d"
server_key_version: "encryptionkey@0"
I20260928 04:47:27.930665 23462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "a8a0bba4a36c4aacb1a9fac99c938485"
format_stamp: "Formatted at 2026-09-28 04:47:27 on dist-test-slave-j5c3"
server_key: "f08f9db25070c1338c7f84d764556615"
server_key_iv: "c2d1d090bad356574b372c4486e7072d"
server_key_version: "encryptionkey@0"
I20260928 04:47:27.932142 23462 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260928 04:47:27.933109 23477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:27.933298 23462 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:47:27.933414 23462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "a8a0bba4a36c4aacb1a9fac99c938485"
format_stamp: "Formatted at 2026-09-28 04:47:27 on dist-test-slave-j5c3"
server_key: "f08f9db25070c1338c7f84d764556615"
server_key_iv: "c2d1d090bad356574b372c4486e7072d"
server_key_version: "encryptionkey@0"
I20260928 04:47:27.933493 23462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:27.951237 23462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:27.953853 23462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:27.954066 23462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:27.958688 23462 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:40465
I20260928 04:47:27.958714 23529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:40465 every 8 connection(s)
I20260928 04:47:27.959131 23462 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:47:27.960075 23530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:27.962376 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 23462
I20260928 04:47:27.962527 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:47:27.962522 23530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8a0bba4a36c4aacb1a9fac99c938485: Bootstrap starting.
I20260928 04:47:27.962726 1931 external_mini_cluster.cc:1428] Setting key daa5b7987a5aeb19a655aefd4e7f4c3f
I20260928 04:47:27.963258 23530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8a0bba4a36c4aacb1a9fac99c938485: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:27.963616 23530 log.cc:824] T 00000000000000000000000000000000 P a8a0bba4a36c4aacb1a9fac99c938485: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:27.964433 23530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8a0bba4a36c4aacb1a9fac99c938485: No bootstrap required, opened a new log
I20260928 04:47:27.966717 23530 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8a0bba4a36c4aacb1a9fac99c938485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a0bba4a36c4aacb1a9fac99c938485" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 40465 } }
I20260928 04:47:27.966918 23530 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8a0bba4a36c4aacb1a9fac99c938485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:27.966953 23530 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8a0bba4a36c4aacb1a9fac99c938485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a0bba4a36c4aacb1a9fac99c938485, State: Initialized, Role: FOLLOWER
I20260928 04:47:27.967068 23530 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8a0bba4a36c4aacb1a9fac99c938485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a0bba4a36c4aacb1a9fac99c938485" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 40465 } }
I20260928 04:47:27.967139 23530 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8a0bba4a36c4aacb1a9fac99c938485 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:47:27.967167 23530 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8a0bba4a36c4aacb1a9fac99c938485 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:47:27.967208 23530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8a0bba4a36c4aacb1a9fac99c938485 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:27.967955 23530 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8a0bba4a36c4aacb1a9fac99c938485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a0bba4a36c4aacb1a9fac99c938485" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 40465 } }
I20260928 04:47:27.968070 23530 leader_election.cc:302] T 00000000000000000000000000000000 P a8a0bba4a36c4aacb1a9fac99c938485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8a0bba4a36c4aacb1a9fac99c938485; no voters:
I20260928 04:47:27.968299 23530 leader_election.cc:288] T 00000000000000000000000000000000 P a8a0bba4a36c4aacb1a9fac99c938485 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:47:27.968319 23535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8a0bba4a36c4aacb1a9fac99c938485 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:27.968453 23535 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8a0bba4a36c4aacb1a9fac99c938485 [term 1 LEADER]: Becoming Leader. State: Replica: a8a0bba4a36c4aacb1a9fac99c938485, State: Running, Role: LEADER
I20260928 04:47:27.968540 23535 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8a0bba4a36c4aacb1a9fac99c938485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a0bba4a36c4aacb1a9fac99c938485" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 40465 } }
I20260928 04:47:27.968603 23530 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8a0bba4a36c4aacb1a9fac99c938485 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:27.968883 23535 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8a0bba4a36c4aacb1a9fac99c938485 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8a0bba4a36c4aacb1a9fac99c938485. Latest consensus state: current_term: 1 leader_uuid: "a8a0bba4a36c4aacb1a9fac99c938485" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a0bba4a36c4aacb1a9fac99c938485" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 40465 } } }
I20260928 04:47:27.969022 23535 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8a0bba4a36c4aacb1a9fac99c938485 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:27.968916 23536 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8a0bba4a36c4aacb1a9fac99c938485 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8a0bba4a36c4aacb1a9fac99c938485" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8a0bba4a36c4aacb1a9fac99c938485" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 40465 } } }
I20260928 04:47:27.969295 23536 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8a0bba4a36c4aacb1a9fac99c938485 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:27.969303 23541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:47:27.969949 23541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 04:47:27 dist-test-slave-j5c3 krb5kdc[23449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:47:27.972023 23541 catalog_manager.cc:1347] Generated new cluster ID: 48bd05e30e8941eb840989e2c5a8f298
I20260928 04:47:27.972079 23541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:47:27.981415 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:27.963362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43993 (local address 127.1.226.254:40465)
0928 04:47:27.963553 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:27.963561 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:27.963587 (+ 26us) server_negotiation.cc:407] Connection header received
0928 04:47:27.964101 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:27.964108 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:27.964217 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:27.964358 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:27.965171 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:27.966140 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:27.967297 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:27.967947 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:27.968065 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:27.971647 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:27.971652 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:27.971653 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:27.971666 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:27.978803 (+ 7137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:27.979421 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:27.979425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:27.979425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:27.979550 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:27.979930 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:27.979931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:27.979931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:27.980368 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:27.980516 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:27.981016 (+ 500us) server_negotiation.cc:299] Negotiation successful
0928 04:47:27.981128 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
I20260928 04:47:27.982565 1931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:47:27.988888 23541 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:47:27.989717 23541 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:47:27.992141 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:27.984134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35407 (local address 127.1.226.254:40465)
0928 04:47:27.984197 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:27.984199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:27.984323 (+ 124us) server_negotiation.cc:407] Connection header received
0928 04:47:27.984372 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:27.984373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:27.984385 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:27.984446 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:27.985120 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:27.985914 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:27.987002 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:27.987309 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:27.987389 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:27.988948 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:27.988950 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:27.988950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:27.988958 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:27.990797 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:27.991254 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:27.991255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:27.991255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:27.991319 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:27.991567 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:27.991568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:27.991568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:27.991703 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:27.991805 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:27.991972 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 04:47:27.992044 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 04:47:27.996119 23541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8a0bba4a36c4aacb1a9fac99c938485: Generated new TSK 0
I20260928 04:47:27.996407 23541 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:47:28.000376 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 23462
W20260928 04:47:28.004703 23447 connection.cc:535] client connection to 127.1.226.254:40465 recv error: Network error: failed to read from TLS socket (remote: 127.1.226.254:40465): Connection reset by peer (error 104)
2026-09-28T04:47:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1364 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:47:28 dist-test-slave-j5c3 krb5kdc[23561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:47:28 dist-test-slave-j5c3 krb5kdc[23561](info): set up 2 sockets
Sep 28 04:47:28 dist-test-slave-j5c3 krb5kdc[23561](info): commencing operation
krb5kdc: starting...
W20260928 04:47:29.048446 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:47:29 dist-test-slave-j5c3 krb5kdc[23561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:47:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:47:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:47:29.173146 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:43229
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:45441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:43229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:29.256353 23574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:29.256560 23574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:29.256584 23574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:29.257712 23574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:29.257747 23574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:29.257761 23574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:29.257774 23574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:29.257786 23574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:29.259239 23574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:45441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:43229
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:43229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:29.259444 23574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:29.259622 23574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:29.264947 23580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:29.265143 23582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:29.264912 23579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:29.265257 23574 server_base.cc:1034] running on GCE node
I20260928 04:47:29.265518 23574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:29.265846 23574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:29.267060 23574 hybrid_clock.cc:648] HybridClock initialized: now 1790570849267035 us; error 39 us; skew 500 ppm
Sep 28 04:47:29 dist-test-slave-j5c3 krb5kdc[23561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570849, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:29.270076 23574 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:47:29.270578 23574 webserver.cc:460] Webserver started at http://127.1.226.254:44061/ using document root <none> and password file <none>
I20260928 04:47:29.270866 23574 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:29.270944 23574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:29.271103 23574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:29.272354 23574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "8f22a00fe5964161bc2ba47a46d525df"
format_stamp: "Formatted at 2026-09-28 04:47:29 on dist-test-slave-j5c3"
server_key: "c1fb8e530dc4983a662a5cf2fc8abfc4"
server_key_iv: "5d50a268c4fd0cf97ae14f014a749bf4"
server_key_version: "encryptionkey@0"
I20260928 04:47:29.272740 23574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "8f22a00fe5964161bc2ba47a46d525df"
format_stamp: "Formatted at 2026-09-28 04:47:29 on dist-test-slave-j5c3"
server_key: "c1fb8e530dc4983a662a5cf2fc8abfc4"
server_key_iv: "5d50a268c4fd0cf97ae14f014a749bf4"
server_key_version: "encryptionkey@0"
I20260928 04:47:29.274251 23574 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:47:29.275099 23589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:29.275338 23574 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:29.275455 23574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "8f22a00fe5964161bc2ba47a46d525df"
format_stamp: "Formatted at 2026-09-28 04:47:29 on dist-test-slave-j5c3"
server_key: "c1fb8e530dc4983a662a5cf2fc8abfc4"
server_key_iv: "5d50a268c4fd0cf97ae14f014a749bf4"
server_key_version: "encryptionkey@0"
I20260928 04:47:29.275535 23574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:29.283350 23574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:29.285492 23574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:29.285656 23574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:29.289932 23574 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:43229
I20260928 04:47:29.289965 23641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:43229 every 8 connection(s)
I20260928 04:47:29.290364 23574 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:47:29.291364 23642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:29.294256 23642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f22a00fe5964161bc2ba47a46d525df: Bootstrap starting.
I20260928 04:47:29.294890 23642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f22a00fe5964161bc2ba47a46d525df: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:29.295148 23642 log.cc:824] T 00000000000000000000000000000000 P 8f22a00fe5964161bc2ba47a46d525df: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:29.296000 23642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f22a00fe5964161bc2ba47a46d525df: No bootstrap required, opened a new log
I20260928 04:47:29.297741 23642 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f22a00fe5964161bc2ba47a46d525df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f22a00fe5964161bc2ba47a46d525df" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 43229 } }
I20260928 04:47:29.297900 23642 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f22a00fe5964161bc2ba47a46d525df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:29.297927 23642 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f22a00fe5964161bc2ba47a46d525df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f22a00fe5964161bc2ba47a46d525df, State: Initialized, Role: FOLLOWER
I20260928 04:47:29.298079 23642 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f22a00fe5964161bc2ba47a46d525df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f22a00fe5964161bc2ba47a46d525df" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 43229 } }
I20260928 04:47:29.298202 23642 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f22a00fe5964161bc2ba47a46d525df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:47:29.298229 23642 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f22a00fe5964161bc2ba47a46d525df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:47:29.298234 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 23574
I20260928 04:47:29.298313 23642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f22a00fe5964161bc2ba47a46d525df [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:29.298372 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:47:29.298554 1931 external_mini_cluster.cc:1428] Setting key ebd1a47927eeb2104c0076d8d6a095ee
I20260928 04:47:29.299657 23642 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f22a00fe5964161bc2ba47a46d525df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f22a00fe5964161bc2ba47a46d525df" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 43229 } }
I20260928 04:47:29.299799 23642 leader_election.cc:302] T 00000000000000000000000000000000 P 8f22a00fe5964161bc2ba47a46d525df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f22a00fe5964161bc2ba47a46d525df; no voters:
I20260928 04:47:29.299993 23642 leader_election.cc:288] T 00000000000000000000000000000000 P 8f22a00fe5964161bc2ba47a46d525df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:47:29.300040 23647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f22a00fe5964161bc2ba47a46d525df [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:29.300302 23647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f22a00fe5964161bc2ba47a46d525df [term 1 LEADER]: Becoming Leader. State: Replica: 8f22a00fe5964161bc2ba47a46d525df, State: Running, Role: LEADER
I20260928 04:47:29.300371 23642 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f22a00fe5964161bc2ba47a46d525df [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:29.300472 23647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f22a00fe5964161bc2ba47a46d525df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f22a00fe5964161bc2ba47a46d525df" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 43229 } }
I20260928 04:47:29.300875 23647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f22a00fe5964161bc2ba47a46d525df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f22a00fe5964161bc2ba47a46d525df. Latest consensus state: current_term: 1 leader_uuid: "8f22a00fe5964161bc2ba47a46d525df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f22a00fe5964161bc2ba47a46d525df" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 43229 } } }
I20260928 04:47:29.300969 23647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f22a00fe5964161bc2ba47a46d525df [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:29.300997 23648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f22a00fe5964161bc2ba47a46d525df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f22a00fe5964161bc2ba47a46d525df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f22a00fe5964161bc2ba47a46d525df" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 43229 } } }
I20260928 04:47:29.301113 23648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f22a00fe5964161bc2ba47a46d525df [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:29.301363 23654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:47:29.301945 23654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:47:29.304744 23664 mvcc.cc:205] Tried to move back new op lower bound from 7334178198745038848 to 7334178198735265792. Current Snapshot: MvccSnapshot[applied={T|T < 7334178198745038848}]
I20260928 04:47:29.305266 23654 catalog_manager.cc:1347] Generated new cluster ID: 790c84684fb646c2a04db915acb73a74
I20260928 04:47:29.305379 23654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:47:29.305518 23646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:29.299112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49715 (local address 127.1.226.254:43229)
0928 04:47:29.299358 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:29.299369 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:29.299410 (+ 41us) server_negotiation.cc:407] Connection header received
0928 04:47:29.299538 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:29.299546 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:29.299609 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:29.299784 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:29.300104 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:29.301679 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:29.302952 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:29.303549 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:29.303673 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:29.304702 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:29.304707 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:29.304708 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:29.304943 (+ 235us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:29.304981 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:29.304993 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:29.305107 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:29.305210 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 04:47:29.305241 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":129,"threads_started":1}
I20260928 04:47:29.306586 1931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:47:29.315686 23646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:29.308233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50203 (local address 127.1.226.254:43229)
0928 04:47:29.308278 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:29.308280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:29.312543 (+ 4263us) server_negotiation.cc:407] Connection header received
0928 04:47:29.312635 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:29.312636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:29.312650 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:29.312832 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:29.312999 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:29.313555 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:29.314352 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:29.314583 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:29.314663 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:29.315412 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:29.315413 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:29.315414 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:29.315419 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:29.315435 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:29.315439 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:29.315493 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:29.315554 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 04:47:29.315566 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:47:29.325654 23654 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:47:29.326243 23654 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:47:29.330281 23654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f22a00fe5964161bc2ba47a46d525df: Generated new TSK 0
I20260928 04:47:29.330497 23654 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:47:29.339558 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 23574
W20260928 04:47:29.343899 23559 connection.cc:535] client connection to 127.1.226.254:43229 recv error: Network error: failed to read from TLS socket (remote: 127.1.226.254:43229): Connection reset by peer (error 104)
2026-09-28T04:47:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:47:29 dist-test-slave-j5c3 krb5kdc[23673](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:47:29 dist-test-slave-j5c3 krb5kdc[23673](info): set up 2 sockets
Sep 28 04:47:29 dist-test-slave-j5c3 krb5kdc[23673](info): commencing operation
krb5kdc: starting...
W20260928 04:47:30.456853 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:47:30 dist-test-slave-j5c3 krb5kdc[23673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:47:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:47:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:47:30.583984 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:38783
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:43309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:38783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:30.668357 23686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:30.668601 23686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:30.668627 23686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:30.669888 23686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:30.669929 23686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:30.669945 23686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:30.669957 23686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:30.669971 23686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:30.671553 23686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:43309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:38783
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:38783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:30.671763 23686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:30.671952 23686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:30.676918 23694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:30.676918 23691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:30.677161 23686 server_base.cc:1034] running on GCE node
W20260928 04:47:30.676925 23692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:30.677536 23686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:30.677870 23686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:30.679014 23686 hybrid_clock.cc:648] HybridClock initialized: now 1790570850679001 us; error 30 us; skew 500 ppm
Sep 28 04:47:30 dist-test-slave-j5c3 krb5kdc[23673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570850, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:30.681629 23686 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:47:30.682013 23686 webserver.cc:460] Webserver started at http://127.1.226.254:33953/ using document root <none> and password file <none>
I20260928 04:47:30.682201 23686 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:30.682241 23686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:30.682327 23686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:30.683321 23686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "82014bbd22d64516935c9d4875da9069"
format_stamp: "Formatted at 2026-09-28 04:47:30 on dist-test-slave-j5c3"
server_key: "003a9dacc2ed983a4f4ab7e1d8c0ea05"
server_key_iv: "88b4ed4183aa0b0a97d642effd5ae9ba"
server_key_version: "encryptionkey@0"
I20260928 04:47:30.683645 23686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "82014bbd22d64516935c9d4875da9069"
format_stamp: "Formatted at 2026-09-28 04:47:30 on dist-test-slave-j5c3"
server_key: "003a9dacc2ed983a4f4ab7e1d8c0ea05"
server_key_iv: "88b4ed4183aa0b0a97d642effd5ae9ba"
server_key_version: "encryptionkey@0"
I20260928 04:47:30.685056 23686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:47:30.685802 23701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:30.686007 23686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:30.686079 23686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "82014bbd22d64516935c9d4875da9069"
format_stamp: "Formatted at 2026-09-28 04:47:30 on dist-test-slave-j5c3"
server_key: "003a9dacc2ed983a4f4ab7e1d8c0ea05"
server_key_iv: "88b4ed4183aa0b0a97d642effd5ae9ba"
server_key_version: "encryptionkey@0"
I20260928 04:47:30.686172 23686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:30.689513 23686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:30.691524 23686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:30.691663 23686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:30.695608 23686 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:38783
I20260928 04:47:30.695663 23753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:38783 every 8 connection(s)
I20260928 04:47:30.695994 23686 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:47:30.697098 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:30.698935 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 23686
I20260928 04:47:30.699074 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:47:30.699278 1931 external_mini_cluster.cc:1428] Setting key 2a10b786e8c7b21065609dcbf2eac02f
I20260928 04:47:30.699962 23754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82014bbd22d64516935c9d4875da9069: Bootstrap starting.
I20260928 04:47:30.700656 23754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82014bbd22d64516935c9d4875da9069: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:30.701081 23754 log.cc:824] T 00000000000000000000000000000000 P 82014bbd22d64516935c9d4875da9069: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:30.702198 23757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:30.699821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58799 (local address 127.1.226.254:38783)
0928 04:47:30.700015 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:30.700026 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:30.700223 (+ 197us) server_negotiation.cc:407] Connection header received
0928 04:47:30.700658 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:30.700665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:30.700691 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:30.700857 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:30.701598 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:30.701603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:30.701603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:30.701608 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:30.701674 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:30.701690 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:30.701774 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:30.701953 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 04:47:30.702008 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":71,"threads_started":1}
I20260928 04:47:30.702201 23754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82014bbd22d64516935c9d4875da9069: No bootstrap required, opened a new log
I20260928 04:47:30.703290 1931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:47:30.705088 23754 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82014bbd22d64516935c9d4875da9069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82014bbd22d64516935c9d4875da9069" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38783 } }
I20260928 04:47:30.705319 23754 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82014bbd22d64516935c9d4875da9069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:30.705375 23754 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82014bbd22d64516935c9d4875da9069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82014bbd22d64516935c9d4875da9069, State: Initialized, Role: FOLLOWER
I20260928 04:47:30.705521 23754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82014bbd22d64516935c9d4875da9069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82014bbd22d64516935c9d4875da9069" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38783 } }
I20260928 04:47:30.705653 23754 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82014bbd22d64516935c9d4875da9069 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:47:30.705741 23754 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82014bbd22d64516935c9d4875da9069 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:47:30.705804 23754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82014bbd22d64516935c9d4875da9069 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:30.706441 23754 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82014bbd22d64516935c9d4875da9069 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82014bbd22d64516935c9d4875da9069" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38783 } }
I20260928 04:47:30.706549 23757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:30.705161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38615 (local address 127.1.226.254:38783)
0928 04:47:30.705207 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:30.705209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:30.705371 (+ 162us) server_negotiation.cc:407] Connection header received
0928 04:47:30.705410 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:30.705411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:30.705416 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:30.705495 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:30.706298 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:30.706299 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:30.706300 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:30.706304 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:30.706319 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:30.706324 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:30.706386 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:30.706450 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 04:47:30.706472 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 04:47:30.706578 23754 leader_election.cc:302] T 00000000000000000000000000000000 P 82014bbd22d64516935c9d4875da9069 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82014bbd22d64516935c9d4875da9069; no voters:
I20260928 04:47:30.706761 23754 leader_election.cc:288] T 00000000000000000000000000000000 P 82014bbd22d64516935c9d4875da9069 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:47:30.706957 23765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82014bbd22d64516935c9d4875da9069 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:30.707078 23754 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82014bbd22d64516935c9d4875da9069 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:30.707144 23765 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82014bbd22d64516935c9d4875da9069 [term 1 LEADER]: Becoming Leader. State: Replica: 82014bbd22d64516935c9d4875da9069, State: Running, Role: LEADER
I20260928 04:47:30.707289 23765 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82014bbd22d64516935c9d4875da9069 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82014bbd22d64516935c9d4875da9069" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38783 } }
I20260928 04:47:30.707737 23766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82014bbd22d64516935c9d4875da9069 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82014bbd22d64516935c9d4875da9069. Latest consensus state: current_term: 1 leader_uuid: "82014bbd22d64516935c9d4875da9069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82014bbd22d64516935c9d4875da9069" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38783 } } }
I20260928 04:47:30.707844 23766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82014bbd22d64516935c9d4875da9069 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:30.708082 23773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:47:30.708217 23764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82014bbd22d64516935c9d4875da9069 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82014bbd22d64516935c9d4875da9069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82014bbd22d64516935c9d4875da9069" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38783 } } }
I20260928 04:47:30.708287 23764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82014bbd22d64516935c9d4875da9069 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:30.708745 23773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:47:30.710419 23773 catalog_manager.cc:1347] Generated new cluster ID: f3e9e8927b5f448ab287aaa1e6dc2c94
I20260928 04:47:30.710470 23773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:47:30.724546 23773 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:47:30.724751 23773 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:47:30.730158 23773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82014bbd22d64516935c9d4875da9069: Generated new TSK 0
I20260928 04:47:30.730383 23773 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:47:30.737371 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 23686
2026-09-28T04:47:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1397 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:47:30 dist-test-slave-j5c3 krb5kdc[23785](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:47:30 dist-test-slave-j5c3 krb5kdc[23785](info): set up 2 sockets
Sep 28 04:47:30 dist-test-slave-j5c3 krb5kdc[23785](info): commencing operation
krb5kdc: starting...
W20260928 04:47:31.844601 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:47:31 dist-test-slave-j5c3 krb5kdc[23785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:47:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:47:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:47:31.971896 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:38265
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:38723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:38265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:32.054363 23798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:32.054579 23798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:32.054600 23798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:32.055761 23798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:32.055806 23798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:32.055821 23798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:32.055835 23798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:32.055846 23798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:32.057451 23798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:38723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:38265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:38265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:32.057713 23798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:32.057958 23798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:32.062955 23804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:32.063019 23806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:32.062947 23803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:32.063249 23798 server_base.cc:1034] running on GCE node
I20260928 04:47:32.063496 23798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:32.063812 23798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:32.064955 23798 hybrid_clock.cc:648] HybridClock initialized: now 1790570852064941 us; error 28 us; skew 500 ppm
Sep 28 04:47:32 dist-test-slave-j5c3 krb5kdc[23785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570852, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:32.072855 23798 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:47:32.073365 23798 webserver.cc:460] Webserver started at http://127.1.226.254:43435/ using document root <none> and password file <none>
I20260928 04:47:32.073611 23798 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:32.073688 23798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:32.073848 23798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:32.074944 23798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "e15b2137e4c043f29b96f196c102c96d"
format_stamp: "Formatted at 2026-09-28 04:47:32 on dist-test-slave-j5c3"
server_key: "97d8cd9f53ebe25d5e1e46b899bc3570"
server_key_iv: "14a516dfae5169019eee26994f848466"
server_key_version: "encryptionkey@0"
I20260928 04:47:32.075320 23798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "e15b2137e4c043f29b96f196c102c96d"
format_stamp: "Formatted at 2026-09-28 04:47:32 on dist-test-slave-j5c3"
server_key: "97d8cd9f53ebe25d5e1e46b899bc3570"
server_key_iv: "14a516dfae5169019eee26994f848466"
server_key_version: "encryptionkey@0"
I20260928 04:47:32.076800 23798 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:47:32.077667 23813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:32.077894 23798 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:47:32.078004 23798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "e15b2137e4c043f29b96f196c102c96d"
format_stamp: "Formatted at 2026-09-28 04:47:32 on dist-test-slave-j5c3"
server_key: "97d8cd9f53ebe25d5e1e46b899bc3570"
server_key_iv: "14a516dfae5169019eee26994f848466"
server_key_version: "encryptionkey@0"
I20260928 04:47:32.078090 23798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:32.091507 23798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:32.093601 23798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:32.093770 23798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:32.098258 23798 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:38265
I20260928 04:47:32.098301 23865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:38265 every 8 connection(s)
I20260928 04:47:32.098675 23798 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:47:32.099560 23866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:32.102064 23866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e15b2137e4c043f29b96f196c102c96d: Bootstrap starting.
I20260928 04:47:32.102617 23866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e15b2137e4c043f29b96f196c102c96d: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:32.102861 23866 log.cc:824] T 00000000000000000000000000000000 P e15b2137e4c043f29b96f196c102c96d: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:32.103556 23866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e15b2137e4c043f29b96f196c102c96d: No bootstrap required, opened a new log
I20260928 04:47:32.105297 23866 raft_consensus.cc:348] T 00000000000000000000000000000000 P e15b2137e4c043f29b96f196c102c96d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15b2137e4c043f29b96f196c102c96d" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38265 } }
I20260928 04:47:32.105440 23866 raft_consensus.cc:374] T 00000000000000000000000000000000 P e15b2137e4c043f29b96f196c102c96d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:32.105464 23866 raft_consensus.cc:729] T 00000000000000000000000000000000 P e15b2137e4c043f29b96f196c102c96d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e15b2137e4c043f29b96f196c102c96d, State: Initialized, Role: FOLLOWER
I20260928 04:47:32.105567 23866 consensus_queue.cc:260] T 00000000000000000000000000000000 P e15b2137e4c043f29b96f196c102c96d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15b2137e4c043f29b96f196c102c96d" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38265 } }
I20260928 04:47:32.105618 23866 raft_consensus.cc:388] T 00000000000000000000000000000000 P e15b2137e4c043f29b96f196c102c96d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:47:32.105638 23866 raft_consensus.cc:482] T 00000000000000000000000000000000 P e15b2137e4c043f29b96f196c102c96d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:47:32.105718 23866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e15b2137e4c043f29b96f196c102c96d [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:32.106588 23866 raft_consensus.cc:504] T 00000000000000000000000000000000 P e15b2137e4c043f29b96f196c102c96d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15b2137e4c043f29b96f196c102c96d" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38265 } }
I20260928 04:47:32.106706 23866 leader_election.cc:302] T 00000000000000000000000000000000 P e15b2137e4c043f29b96f196c102c96d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e15b2137e4c043f29b96f196c102c96d; no voters:
I20260928 04:47:32.106902 23866 leader_election.cc:288] T 00000000000000000000000000000000 P e15b2137e4c043f29b96f196c102c96d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:47:32.106945 23869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e15b2137e4c043f29b96f196c102c96d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:32.107105 23869 raft_consensus.cc:686] T 00000000000000000000000000000000 P e15b2137e4c043f29b96f196c102c96d [term 1 LEADER]: Becoming Leader. State: Replica: e15b2137e4c043f29b96f196c102c96d, State: Running, Role: LEADER
I20260928 04:47:32.107218 23866 sys_catalog.cc:564] T 00000000000000000000000000000000 P e15b2137e4c043f29b96f196c102c96d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:32.107208 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 23798
I20260928 04:47:32.107237 23869 consensus_queue.cc:237] T 00000000000000000000000000000000 P e15b2137e4c043f29b96f196c102c96d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15b2137e4c043f29b96f196c102c96d" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38265 } }
I20260928 04:47:32.107329 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:47:32.107533 1931 external_mini_cluster.cc:1428] Setting key bdf2e7b579c1c87774346c92b3961f5a
I20260928 04:47:32.107707 23871 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15b2137e4c043f29b96f196c102c96d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e15b2137e4c043f29b96f196c102c96d. Latest consensus state: current_term: 1 leader_uuid: "e15b2137e4c043f29b96f196c102c96d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15b2137e4c043f29b96f196c102c96d" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38265 } } }
I20260928 04:47:32.107789 23871 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15b2137e4c043f29b96f196c102c96d [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:32.107987 23870 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15b2137e4c043f29b96f196c102c96d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e15b2137e4c043f29b96f196c102c96d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15b2137e4c043f29b96f196c102c96d" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 38265 } } }
I20260928 04:47:32.108050 23870 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15b2137e4c043f29b96f196c102c96d [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:32.108425 23875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:47:32.109025 23875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:47:32.110656 23875 catalog_manager.cc:1347] Generated new cluster ID: f4b8e7f56e03455d8e2391b08530ffa1
I20260928 04:47:32.110711 23875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:47:32 dist-test-slave-j5c3 krb5kdc[23785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:47:32.118794 23880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:32.108220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40519 (local address 127.1.226.254:38265)
0928 04:47:32.108429 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:32.108439 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:32.108474 (+ 35us) server_negotiation.cc:407] Connection header received
0928 04:47:32.108601 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:32.108608 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:32.108670 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:32.109750 (+ 1080us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:32.109863 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:32.111060 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:32.111903 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:32.112419 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:32.112533 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:32.114926 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:32.114930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:32.114930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:32.114943 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:32.117188 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:32.117572 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:32.117576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:32.117576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:32.117684 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:32.117895 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:32.117896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:32.117897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:32.118145 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:32.118322 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:32.118423 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 04:47:32.118568 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260928 04:47:32.119736 1931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:47:32.127133 23880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:32.121145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43305 (local address 127.1.226.254:38265)
0928 04:47:32.121184 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:32.121185 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:32.121325 (+ 140us) server_negotiation.cc:407] Connection header received
0928 04:47:32.121399 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:32.121400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:32.121411 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:32.121459 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:32.121957 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:32.122440 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:32.123122 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:32.123374 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:32.123453 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:32.124661 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:32.124663 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:32.124663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:32.124670 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:32.126027 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:32.126381 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:32.126382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:32.126383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:32.126440 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:32.126628 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:32.126629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:32.126630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:32.126746 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:32.126848 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:32.126946 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 04:47:32.127022 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:47:32.133776 23875 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:47:32.134617 23875 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:47:32.139005 23875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e15b2137e4c043f29b96f196c102c96d: Generated new TSK 0
I20260928 04:47:32.139214 23875 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:47:32.145097 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 23798
2026-09-28T04:47:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:47:32 dist-test-slave-j5c3 krb5kdc[23897](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:47:32 dist-test-slave-j5c3 krb5kdc[23897](info): set up 2 sockets
Sep 28 04:47:32 dist-test-slave-j5c3 krb5kdc[23897](info): commencing operation
krb5kdc: starting...
W20260928 04:47:33.188444 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:47:33 dist-test-slave-j5c3 krb5kdc[23897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:47:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:47:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:47:33.315553 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:40461
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:45093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:40461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:33.396559 23910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:33.396773 23910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:33.396795 23910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:33.397997 23910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:33.398043 23910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:33.398057 23910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:33.398072 23910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:33.398083 23910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:33.399607 23910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:45093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:40461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:40461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:33.399868 23910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:33.400100 23910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:47:33.405143 23910 server_base.cc:1034] running on GCE node
W20260928 04:47:33.405128 23915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:33.405117 23916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:33.405128 23918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:33.405593 23910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:33.405900 23910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:33.407065 23910 hybrid_clock.cc:648] HybridClock initialized: now 1790570853407034 us; error 42 us; skew 500 ppm
Sep 28 04:47:33 dist-test-slave-j5c3 krb5kdc[23897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570853, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:33.413811 23910 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:47:33.414284 23910 webserver.cc:460] Webserver started at http://127.1.226.254:46371/ using document root <none> and password file <none>
I20260928 04:47:33.414512 23910 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:33.414559 23910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:33.414728 23910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:33.415868 23910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "c08738bde77f4f21ac6dd71d810cbc93"
format_stamp: "Formatted at 2026-09-28 04:47:33 on dist-test-slave-j5c3"
server_key: "202d47ade9e045bccf25951cbced43f7"
server_key_iv: "ea2b739e14fce9d0dadf7ecc23fc3a9b"
server_key_version: "encryptionkey@0"
I20260928 04:47:33.416275 23910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "c08738bde77f4f21ac6dd71d810cbc93"
format_stamp: "Formatted at 2026-09-28 04:47:33 on dist-test-slave-j5c3"
server_key: "202d47ade9e045bccf25951cbced43f7"
server_key_iv: "ea2b739e14fce9d0dadf7ecc23fc3a9b"
server_key_version: "encryptionkey@0"
I20260928 04:47:33.417721 23910 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:33.418576 23925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:33.418803 23910 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:33.418908 23910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "c08738bde77f4f21ac6dd71d810cbc93"
format_stamp: "Formatted at 2026-09-28 04:47:33 on dist-test-slave-j5c3"
server_key: "202d47ade9e045bccf25951cbced43f7"
server_key_iv: "ea2b739e14fce9d0dadf7ecc23fc3a9b"
server_key_version: "encryptionkey@0"
I20260928 04:47:33.418987 23910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:33.443400 23910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:33.445549 23910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:33.445750 23910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:33.449987 23910 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:40461
I20260928 04:47:33.450030 23977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:40461 every 8 connection(s)
I20260928 04:47:33.450426 23910 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:47:33.450809 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 23910
I20260928 04:47:33.450929 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:47:33.451109 1931 external_mini_cluster.cc:1428] Setting key 0a076d87c3ca6f96e50fbf3696c769dd
I20260928 04:47:33.451507 23978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:33.454802 23978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c08738bde77f4f21ac6dd71d810cbc93: Bootstrap starting.
I20260928 04:47:33.455466 23978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c08738bde77f4f21ac6dd71d810cbc93: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:33.455767 23978 log.cc:824] T 00000000000000000000000000000000 P c08738bde77f4f21ac6dd71d810cbc93: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:33.456699 23978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c08738bde77f4f21ac6dd71d810cbc93: No bootstrap required, opened a new log
I20260928 04:47:33.456882 23981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:33.451675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50035 (local address 127.1.226.254:40461)
0928 04:47:33.451857 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:33.451865 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:33.451891 (+ 26us) server_negotiation.cc:407] Connection header received
0928 04:47:33.452376 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:33.452383 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:33.452432 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:33.452548 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:33.452791 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:33.453644 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:33.454816 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:33.455274 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:33.455357 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:33.456085 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:33.456089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:33.456089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:33.456330 (+ 241us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:33.456376 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:33.456387 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:33.456439 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:33.456541 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 04:47:33.456595 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
I20260928 04:47:33.458045 1931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:47:33.458899 23978 raft_consensus.cc:348] T 00000000000000000000000000000000 P c08738bde77f4f21ac6dd71d810cbc93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08738bde77f4f21ac6dd71d810cbc93" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 40461 } }
I20260928 04:47:33.459084 23978 raft_consensus.cc:374] T 00000000000000000000000000000000 P c08738bde77f4f21ac6dd71d810cbc93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:33.459151 23978 raft_consensus.cc:729] T 00000000000000000000000000000000 P c08738bde77f4f21ac6dd71d810cbc93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c08738bde77f4f21ac6dd71d810cbc93, State: Initialized, Role: FOLLOWER
I20260928 04:47:33.459268 23978 consensus_queue.cc:260] T 00000000000000000000000000000000 P c08738bde77f4f21ac6dd71d810cbc93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08738bde77f4f21ac6dd71d810cbc93" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 40461 } }
I20260928 04:47:33.459339 23978 raft_consensus.cc:388] T 00000000000000000000000000000000 P c08738bde77f4f21ac6dd71d810cbc93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:47:33.459360 23978 raft_consensus.cc:482] T 00000000000000000000000000000000 P c08738bde77f4f21ac6dd71d810cbc93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:47:33.459435 23978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c08738bde77f4f21ac6dd71d810cbc93 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:33.459991 23978 raft_consensus.cc:504] T 00000000000000000000000000000000 P c08738bde77f4f21ac6dd71d810cbc93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08738bde77f4f21ac6dd71d810cbc93" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 40461 } }
I20260928 04:47:33.460078 23978 leader_election.cc:302] T 00000000000000000000000000000000 P c08738bde77f4f21ac6dd71d810cbc93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c08738bde77f4f21ac6dd71d810cbc93; no voters:
I20260928 04:47:33.460282 23978 leader_election.cc:288] T 00000000000000000000000000000000 P c08738bde77f4f21ac6dd71d810cbc93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:47:33.460315 23987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c08738bde77f4f21ac6dd71d810cbc93 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:33.460426 23987 raft_consensus.cc:686] T 00000000000000000000000000000000 P c08738bde77f4f21ac6dd71d810cbc93 [term 1 LEADER]: Becoming Leader. State: Replica: c08738bde77f4f21ac6dd71d810cbc93, State: Running, Role: LEADER
I20260928 04:47:33.460521 23987 consensus_queue.cc:237] T 00000000000000000000000000000000 P c08738bde77f4f21ac6dd71d810cbc93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08738bde77f4f21ac6dd71d810cbc93" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 40461 } }
I20260928 04:47:33.460556 23978 sys_catalog.cc:564] T 00000000000000000000000000000000 P c08738bde77f4f21ac6dd71d810cbc93 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:33.460824 23987 sys_catalog.cc:455] T 00000000000000000000000000000000 P c08738bde77f4f21ac6dd71d810cbc93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c08738bde77f4f21ac6dd71d810cbc93. Latest consensus state: current_term: 1 leader_uuid: "c08738bde77f4f21ac6dd71d810cbc93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08738bde77f4f21ac6dd71d810cbc93" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 40461 } } }
I20260928 04:47:33.460908 23987 sys_catalog.cc:458] T 00000000000000000000000000000000 P c08738bde77f4f21ac6dd71d810cbc93 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:33.460839 23988 sys_catalog.cc:455] T 00000000000000000000000000000000 P c08738bde77f4f21ac6dd71d810cbc93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c08738bde77f4f21ac6dd71d810cbc93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08738bde77f4f21ac6dd71d810cbc93" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 40461 } } }
I20260928 04:47:33.461104 23988 sys_catalog.cc:458] T 00000000000000000000000000000000 P c08738bde77f4f21ac6dd71d810cbc93 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:33.461120 23993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:47:33.462097 23993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:47:33.464350 23993 catalog_manager.cc:1347] Generated new cluster ID: 80c1a014f70d46b2b5248ec2d1c8c92b
I20260928 04:47:33.464406 23993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:47:33.464547 23981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:33.460501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53195 (local address 127.1.226.254:40461)
0928 04:47:33.460546 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:33.460548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:33.460812 (+ 264us) server_negotiation.cc:407] Connection header received
0928 04:47:33.460921 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:33.460923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:33.460936 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:33.460980 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:33.461251 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:33.461974 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:33.462942 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:33.463246 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:33.463321 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:33.464109 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:33.464111 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:33.464111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:33.464115 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:33.464131 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:33.464135 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:33.464214 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:33.464453 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 04:47:33.464478 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:47:33.490923 23993 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:47:33.491709 23993 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:47:33.497262 23993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c08738bde77f4f21ac6dd71d810cbc93: Generated new TSK 0
I20260928 04:47:33.497469 23993 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:47:33.509141 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 23910
2026-09-28T04:47:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1361 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:47:33 dist-test-slave-j5c3 krb5kdc[24009](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:47:33 dist-test-slave-j5c3 krb5kdc[24009](info): set up 2 sockets
Sep 28 04:47:33 dist-test-slave-j5c3 krb5kdc[24009](info): commencing operation
krb5kdc: starting...
W20260928 04:47:34.576493 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:47:34 dist-test-slave-j5c3 krb5kdc[24009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:47:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:47:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:47:34.695783 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:44155
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:45861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:44155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:34.778949 24022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:34.779174 24022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:34.779201 24022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:34.780519 24022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:34.780573 24022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:34.780588 24022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:34.780601 24022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:34.780642 24022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:34.782236 24022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:45861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:44155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:44155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:34.782507 24022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:34.782766 24022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:34.787846 24027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:34.787926 24022 server_base.cc:1034] running on GCE node
W20260928 04:47:34.787868 24028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:34.787868 24030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:34.788314 24022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:34.788571 24022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:34.789740 24022 hybrid_clock.cc:648] HybridClock initialized: now 1790570854789734 us; error 36 us; skew 500 ppm
Sep 28 04:47:34 dist-test-slave-j5c3 krb5kdc[24009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570854, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:34.792259 24022 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:47:34.792704 24022 webserver.cc:460] Webserver started at http://127.1.226.254:43883/ using document root <none> and password file <none>
I20260928 04:47:34.792961 24022 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:34.793032 24022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:34.793176 24022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:34.794230 24022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "a2a716232ee143c09f1e11ce319e68a8"
format_stamp: "Formatted at 2026-09-28 04:47:34 on dist-test-slave-j5c3"
server_key: "9f9701cfbcd7f9ae2a70bf2a9a5afa48"
server_key_iv: "aea1b0653f6a1fcc1094201f056f057e"
server_key_version: "encryptionkey@0"
I20260928 04:47:34.794600 24022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "a2a716232ee143c09f1e11ce319e68a8"
format_stamp: "Formatted at 2026-09-28 04:47:34 on dist-test-slave-j5c3"
server_key: "9f9701cfbcd7f9ae2a70bf2a9a5afa48"
server_key_iv: "aea1b0653f6a1fcc1094201f056f057e"
server_key_version: "encryptionkey@0"
I20260928 04:47:34.795953 24022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 04:47:34.796792 24037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:34.797000 24022 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:47:34.797104 24022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "a2a716232ee143c09f1e11ce319e68a8"
format_stamp: "Formatted at 2026-09-28 04:47:34 on dist-test-slave-j5c3"
server_key: "9f9701cfbcd7f9ae2a70bf2a9a5afa48"
server_key_iv: "aea1b0653f6a1fcc1094201f056f057e"
server_key_version: "encryptionkey@0"
I20260928 04:47:34.797186 24022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:34.800496 24022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:34.802517 24022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:34.802677 24022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:34.806708 24022 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:44155
I20260928 04:47:34.806742 24089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:44155 every 8 connection(s)
I20260928 04:47:34.807086 24022 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:47:34.808197 24090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:34.810551 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 24022
I20260928 04:47:34.810669 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:47:34.810698 24090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2a716232ee143c09f1e11ce319e68a8: Bootstrap starting.
I20260928 04:47:34.810869 1931 external_mini_cluster.cc:1428] Setting key b5bd2be596fdd384005a9500b070d062
I20260928 04:47:34.811432 24090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2a716232ee143c09f1e11ce319e68a8: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:34.811769 24090 log.cc:824] T 00000000000000000000000000000000 P a2a716232ee143c09f1e11ce319e68a8: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:34.812547 24090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2a716232ee143c09f1e11ce319e68a8: No bootstrap required, opened a new log
I20260928 04:47:34.813712 24093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:34.811414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51149 (local address 127.1.226.254:44155)
0928 04:47:34.811721 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:34.811731 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:34.811763 (+ 32us) server_negotiation.cc:407] Connection header received
0928 04:47:34.812406 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:34.812414 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:34.812450 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:34.812598 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:34.813285 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:34.813289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:34.813289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:34.813294 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:34.813340 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:34.813353 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:34.813442 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:34.813481 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 04:47:34.813520 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":82,"threads_started":1}
I20260928 04:47:34.814777 1931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:47:34.815038 24090 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2a716232ee143c09f1e11ce319e68a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a716232ee143c09f1e11ce319e68a8" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44155 } }
I20260928 04:47:34.815201 24090 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2a716232ee143c09f1e11ce319e68a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:47:34.815240 24090 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2a716232ee143c09f1e11ce319e68a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a716232ee143c09f1e11ce319e68a8, State: Initialized, Role: FOLLOWER
I20260928 04:47:34.815380 24090 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2a716232ee143c09f1e11ce319e68a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a716232ee143c09f1e11ce319e68a8" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44155 } }
I20260928 04:47:34.815471 24090 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2a716232ee143c09f1e11ce319e68a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:47:34.815506 24090 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2a716232ee143c09f1e11ce319e68a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:47:34.815565 24090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2a716232ee143c09f1e11ce319e68a8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:34.816394 24090 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2a716232ee143c09f1e11ce319e68a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a716232ee143c09f1e11ce319e68a8" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44155 } }
I20260928 04:47:34.816510 24090 leader_election.cc:302] T 00000000000000000000000000000000 P a2a716232ee143c09f1e11ce319e68a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2a716232ee143c09f1e11ce319e68a8; no voters:
I20260928 04:47:34.816669 24090 leader_election.cc:288] T 00000000000000000000000000000000 P a2a716232ee143c09f1e11ce319e68a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:47:34.816705 24100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2a716232ee143c09f1e11ce319e68a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:34.817004 24090 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2a716232ee143c09f1e11ce319e68a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:34.817230 24100 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2a716232ee143c09f1e11ce319e68a8 [term 1 LEADER]: Becoming Leader. State: Replica: a2a716232ee143c09f1e11ce319e68a8, State: Running, Role: LEADER
I20260928 04:47:34.817399 24100 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2a716232ee143c09f1e11ce319e68a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a716232ee143c09f1e11ce319e68a8" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44155 } }
I20260928 04:47:34.817552 24093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:34.816448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52783 (local address 127.1.226.254:44155)
0928 04:47:34.816500 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:34.816502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:34.816567 (+ 65us) server_negotiation.cc:407] Connection header received
0928 04:47:34.816633 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:34.816634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:34.816639 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:34.816712 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:34.817297 (+ 585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:34.817299 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:34.817299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:47:34.817304 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:34.817318 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:47:34.817323 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:34.817354 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:34.817454 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 04:47:34.817469 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 04:47:34.818547 24102 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2a716232ee143c09f1e11ce319e68a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2a716232ee143c09f1e11ce319e68a8. Latest consensus state: current_term: 1 leader_uuid: "a2a716232ee143c09f1e11ce319e68a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a716232ee143c09f1e11ce319e68a8" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44155 } } }
I20260928 04:47:34.818611 24102 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2a716232ee143c09f1e11ce319e68a8 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:34.818377 24101 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2a716232ee143c09f1e11ce319e68a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2a716232ee143c09f1e11ce319e68a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a716232ee143c09f1e11ce319e68a8" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 44155 } } }
I20260928 04:47:34.818835 24101 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2a716232ee143c09f1e11ce319e68a8 [sys.catalog]: This master's current role is: LEADER
W20260928 04:47:34.818881 24114 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2a716232ee143c09f1e11ce319e68a8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:47:34.818944 24114 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:47:34.822353 24116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:47:34.822604 24116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:47:34.824471 24116 catalog_manager.cc:1347] Generated new cluster ID: 1f08b4dc78654f68ad478cc17649ac32
I20260928 04:47:34.824517 24116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:47:34.861202 24116 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:47:34.861431 24116 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:47:34.870299 24116 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2a716232ee143c09f1e11ce319e68a8: Generated new TSK 0
I20260928 04:47:34.870559 24116 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:47:34.879868 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 24022
2026-09-28T04:47:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1371 ms)
[----------] 18 tests from AuthTokenIssuingTest (24508 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:47:34 dist-test-slave-j5c3 krb5kdc[24121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:47:34 dist-test-slave-j5c3 krb5kdc[24121](info): set up 2 sockets
Sep 28 04:47:34 dist-test-slave-j5c3 krb5kdc[24121](info): commencing operation
krb5kdc: starting...
W20260928 04:47:35.916440 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:47:36 dist-test-slave-j5c3 krb5kdc[24121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:47:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:47:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:47:36.042416 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:45325
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:42185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:45325,127.1.226.253:35891,127.1.226.252:37119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:36.123943 24134 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 04:47:36.124202 24134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:36.124220 24134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:36.124243 24134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:36.125465 24134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:36.125511 24134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:36.125526 24134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:36.125540 24134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:36.125591 24134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:36.127113 24134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:42185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:45325,127.1.226.253:35891,127.1.226.252:37119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:45325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:36.127378 24134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:36.127611 24134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:47:36.132666 24134 server_base.cc:1034] running on GCE node
W20260928 04:47:36.132650 24140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:36.132650 24142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:36.132722 24139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:36.133199 24134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:36.133522 24134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:36.134703 24134 hybrid_clock.cc:648] HybridClock initialized: now 1790570856134695 us; error 32 us; skew 500 ppm
Sep 28 04:47:36 dist-test-slave-j5c3 krb5kdc[24121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570856, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:36.137534 24134 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:47:36.138047 24134 webserver.cc:460] Webserver started at http://127.1.226.254:45703/ using document root <none> and password file <none>
I20260928 04:47:36.138284 24134 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:36.138353 24134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:36.138504 24134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:36.139575 24134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "573c813bd87b44d9848cb7a2fcc81e44"
format_stamp: "Formatted at 2026-09-28 04:47:36 on dist-test-slave-j5c3"
server_key: "67c49e2c8170ee263c94c4232ffaf746"
server_key_iv: "89d17e05503a4dc1b2529f267dea830e"
server_key_version: "encryptionkey@0"
I20260928 04:47:36.139964 24134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "573c813bd87b44d9848cb7a2fcc81e44"
format_stamp: "Formatted at 2026-09-28 04:47:36 on dist-test-slave-j5c3"
server_key: "67c49e2c8170ee263c94c4232ffaf746"
server_key_iv: "89d17e05503a4dc1b2529f267dea830e"
server_key_version: "encryptionkey@0"
I20260928 04:47:36.141472 24134 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:36.142297 24149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:36.142479 24134 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:36.142583 24134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "573c813bd87b44d9848cb7a2fcc81e44"
format_stamp: "Formatted at 2026-09-28 04:47:36 on dist-test-slave-j5c3"
server_key: "67c49e2c8170ee263c94c4232ffaf746"
server_key_iv: "89d17e05503a4dc1b2529f267dea830e"
server_key_version: "encryptionkey@0"
I20260928 04:47:36.142673 24134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:36.154912 24134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:36.157472 24134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:36.157646 24134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:36.161794 24134 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:45325
I20260928 04:47:36.161844 24201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:45325 every 8 connection(s)
I20260928 04:47:36.162248 24134 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:47:36.163396 24202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:36.165118 24202 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45325 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:47:36.167150 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 24134
I20260928 04:47:36.167299 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:47:36.167475 1931 external_mini_cluster.cc:1428] Setting key 4deeb406ab5ac40c16beee0905d0dd6c
Sep 28 04:47:36 dist-test-slave-j5c3 krb5kdc[24121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570856, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.226.253@KRBTEST.COM; defaulting to no policy
I20260928 04:47:36.177379 24204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.165873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:45325 (local address 127.0.0.1:37298)
0928 04:47:36.166082 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:36.166122 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:36.166609 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:36.166939 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:36.166942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:36.167190 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:36.167446 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:36.167452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.169092 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.169094 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:36.170312 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:36.170315 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.170836 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.170837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:36.170956 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:36.171595 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:47:36.171607 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:36.173709 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:36.175629 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:36.175638 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:36.175641 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:36.175924 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:36.176221 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:36.176222 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:36.176222 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:36.176338 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:36.176819 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:36.176826 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:36.176891 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 04:47:36.176979 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 04:47:36.177118 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":96,"threads_started":1}
I20260928 04:47:36.177388 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.165902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37298 (local address 127.1.226.254:45325)
0928 04:47:36.166130 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:36.166134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:36.166202 (+ 68us) server_negotiation.cc:407] Connection header received
0928 04:47:36.166799 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:36.166802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:36.166820 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:36.166908 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:36.167587 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.168975 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.170402 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.170708 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.170801 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:36.173885 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:36.173892 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:36.173892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:36.173898 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:36.175530 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.176014 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.176017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.176017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.176096 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.176430 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.176431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.176432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.176670 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:36.176746 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:36.177036 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 04:47:36.177192 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":206,"thread_start_us":85,"threads_started":1}
I20260928 04:47:36.178311 24202 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.253" port: 35891 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:47:36.178823 24204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.178602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:35891 (local address 127.0.0.1:34772)
0928 04:47:36.178671 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:36.178721 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.253:35891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 04:47:36.179049 24151 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.226.253:35891: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.253@KRBTEST.COM" created.
I20260928 04:47:36.179533 24204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.179432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:35891 (local address 127.0.0.1:34788)
0928 04:47:36.179480 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:36.179514 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.253:35891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 04:47:36.179667 24202 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.226.253:35891: Network error: Client connection negotiation failed: client connection to 127.1.226.253:35891: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.253.keytab.
Entry for principal kudu/127.1.226.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.253.keytab.
I20260928 04:47:36.193260 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.253:35891
--webserver_interface=127.1.226.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.253.keytab
--principal=kudu/127.1.226.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:42185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:45325,127.1.226.253:35891,127.1.226.252:37119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:47:36.228917 24202 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.226.253" port: 35891 } attempt: 1
I20260928 04:47:36.229456 24204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.229291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:35891 (local address 127.0.0.1:34796)
0928 04:47:36.229377 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:36.229412 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.253:35891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 04:47:36.229861 24204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.229715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:35891 (local address 127.0.0.1:34806)
0928 04:47:36.229808 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:36.229835 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.253:35891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260928 04:47:36.229969 24202 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.226.253:35891: Network error: Client connection negotiation failed: client connection to 127.1.226.253:35891: connect: Connection refused (error 111)
W20260928 04:47:36.276362 24209 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 04:47:36.276619 24209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:36.276648 24209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:36.276690 24209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:36.277932 24209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:36.277995 24209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:36.278019 24209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:36.278046 24209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:36.278076 24209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:36.279606 24209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:42185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:45325,127.1.226.253:35891,127.1.226.252:37119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.253.keytab
--principal=kudu/127.1.226.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.253:35891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.226.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:36.279896 24209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:36.280129 24209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:36.285116 24217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:36.285116 24215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:36.285205 24209 server_base.cc:1034] running on GCE node
W20260928 04:47:36.285133 24214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:36.285652 24209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:36.285943 24209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:36.287086 24209 hybrid_clock.cc:648] HybridClock initialized: now 1790570856287059 us; error 45 us; skew 500 ppm
Sep 28 04:47:36 dist-test-slave-j5c3 krb5kdc[24121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570856, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:36.289722 24209 init.cc:377] Logged in from keytab as kudu/127.1.226.253@KRBTEST.COM (short username kudu)
I20260928 04:47:36.290112 24209 webserver.cc:460] Webserver started at http://127.1.226.253:37493/ using document root <none> and password file <none>
I20260928 04:47:36.290293 24209 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:36.290341 24209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:36.290432 24209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:36.291460 24209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-1/data/instance:
uuid: "5da39d55d9b94877ac32dddea369a852"
format_stamp: "Formatted at 2026-09-28 04:47:36 on dist-test-slave-j5c3"
server_key: "5ab229d3bd3e0457821ccc2b6e9592ff"
server_key_iv: "8feeb9bc11f0c007cab2a11b1806f732"
server_key_version: "encryptionkey@0"
I20260928 04:47:36.291764 24209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-1/wal/instance:
uuid: "5da39d55d9b94877ac32dddea369a852"
format_stamp: "Formatted at 2026-09-28 04:47:36 on dist-test-slave-j5c3"
server_key: "5ab229d3bd3e0457821ccc2b6e9592ff"
server_key_iv: "8feeb9bc11f0c007cab2a11b1806f732"
server_key_version: "encryptionkey@0"
I20260928 04:47:36.293181 24209 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:47:36.293969 24224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:36.294168 24209 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:36.294258 24209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-1/wal
uuid: "5da39d55d9b94877ac32dddea369a852"
format_stamp: "Formatted at 2026-09-28 04:47:36 on dist-test-slave-j5c3"
server_key: "5ab229d3bd3e0457821ccc2b6e9592ff"
server_key_iv: "8feeb9bc11f0c007cab2a11b1806f732"
server_key_version: "encryptionkey@0"
I20260928 04:47:36.294322 24209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:36.298111 24202 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.226.253" port: 35891 } attempt: 2
I20260928 04:47:36.298555 24204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.298419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:35891 (local address 127.0.0.1:34812)
0928 04:47:36.298484 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:36.298517 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.253:35891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 04:47:36.298952 24204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.298861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:35891 (local address 127.0.0.1:34814)
0928 04:47:36.298911 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:36.298937 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.253:35891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 04:47:36.299077 24202 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.226.253:35891: Network error: Client connection negotiation failed: client connection to 127.1.226.253:35891: connect: Connection refused (error 111)
I20260928 04:47:36.307094 24209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:36.309329 24209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:36.309480 24209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:36.314522 24209 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.253:35891
I20260928 04:47:36.314551 24276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.253:35891 every 8 connection(s)
I20260928 04:47:36.314972 24209 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-1/data/info.pb
I20260928 04:47:36.316092 24277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:36.317656 24277 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45325 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:47:36.318610 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 24209
I20260928 04:47:36.318713 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-1/wal/instance
I20260928 04:47:36.318881 1931 external_mini_cluster.cc:1428] Setting key 709803f997142e7da836e60144bfb8d5
Sep 28 04:47:36 dist-test-slave-j5c3 krb5kdc[24121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570856, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.253@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.226.252@KRBTEST.COM; defaulting to no policy
I20260928 04:47:36.328209 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.318373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.1.226.254:45325)
0928 04:47:36.318478 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:36.318481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:36.318735 (+ 254us) server_negotiation.cc:407] Connection header received
0928 04:47:36.319440 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:36.319441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:36.319455 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:36.319510 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:36.320492 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.321271 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.322268 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.322615 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.322690 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:36.324966 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:36.324967 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:36.324968 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:36.324976 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:36.326572 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.327033 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.327035 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.327035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.327099 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.327400 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.327400 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.327401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.327511 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:36.327566 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:36.328009 (+ 443us) server_negotiation.cc:299] Negotiation successful
0928 04:47:36.328080 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 04:47:36.328428 24279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.318293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:45325 (local address 127.0.0.1:37308)
0928 04:47:36.318570 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:36.318622 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:36.319328 (+ 706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:36.319582 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:36.319586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:36.320006 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:36.320357 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:36.320365 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.321370 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.321371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:36.322177 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:36.322179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.322720 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.322721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:36.322861 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:36.323433 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:47:36.323442 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:36.324867 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:36.326657 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:36.326663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:36.326664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:36.326958 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:36.327210 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:36.327211 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:36.327212 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:36.327327 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:36.327627 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:36.327634 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:36.327879 (+ 245us) client_negotiation.cc:769] Sending connection context
0928 04:47:36.327980 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 04:47:36.328107 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":103,"threads_started":1}
I20260928 04:47:36.329100 24277 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.253" port: 35891 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.252@KRBTEST.COM" created.
Sep 28 04:47:36 dist-test-slave-j5c3 krb5kdc[24121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570856, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.253@KRBTEST.COM for kudu/127.1.226.253@KRBTEST.COM
I20260928 04:47:36.337809 24279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.329407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:35891 (local address 127.0.0.1:34826)
0928 04:47:36.329655 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:36.329667 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:36.329718 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:36.329962 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:36.329962 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:36.330076 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:36.330130 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:36.330131 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.331322 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.331323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:36.332234 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:36.332237 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.332690 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.332691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:36.332773 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:36.333173 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:47:36.333179 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:36.334738 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:36.336708 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:36.336710 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:36.336710 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:36.336944 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:36.337185 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:36.337185 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:36.337186 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:36.337260 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:36.337545 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:36.337546 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:36.337598 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 04:47:36.337657 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 04:47:36.337740 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233}
I20260928 04:47:36.337998 24281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.329485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34826 (local address 127.1.226.253:35891)
0928 04:47:36.329734 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:36.329740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:36.329768 (+ 28us) server_negotiation.cc:407] Connection header received
0928 04:47:36.329793 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:36.329796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:36.329811 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:36.329934 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:36.330201 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.331243 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.332311 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.332614 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.332669 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:36.334843 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:36.334850 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:36.334850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:36.334857 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:36.336600 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.337049 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.337052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.337053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.337110 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.337331 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.337331 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.337332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.337474 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:36.337534 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:36.337686 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 04:47:36.337799 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":209,"thread_start_us":115,"threads_started":1}
I20260928 04:47:36.338668 24277 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.252" port: 37119 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:47:36.339088 24279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.338964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:37119 (local address 127.0.0.1:34312)
0928 04:47:36.339010 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:36.339070 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 04:47:36.339260 24225 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.226.252:37119: connect: Connection refused (error 111)
I20260928 04:47:36.339697 24279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.339611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:37119 (local address 127.0.0.1:34314)
0928 04:47:36.339655 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:36.339680 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 04:47:36.339855 24277 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.226.252:37119: Network error: Client connection negotiation failed: client connection to 127.1.226.252:37119: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.252.keytab.
Entry for principal kudu/127.1.226.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.252.keytab.
I20260928 04:47:36.346556 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.252:37119
--webserver_interface=127.1.226.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.252.keytab
--principal=kudu/127.1.226.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:42185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:45325,127.1.226.253:35891,127.1.226.252:37119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:47:36.389081 24277 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.226.252" port: 37119 } attempt: 1
I20260928 04:47:36.389690 24279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.389532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:37119 (local address 127.0.0.1:34326)
0928 04:47:36.389611 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:36.389659 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 04:47:36.390096 24279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.390005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:37119 (local address 127.0.0.1:34336)
0928 04:47:36.390050 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:36.390077 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 04:47:36.390213 24202 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.226.253" port: 35891 } attempt: 3
W20260928 04:47:36.390239 24277 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.226.252:37119: Network error: Client connection negotiation failed: client connection to 127.1.226.252:37119: connect: Connection refused (error 111)
Sep 28 04:47:36 dist-test-slave-j5c3 krb5kdc[24121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570856, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for kudu/127.1.226.253@KRBTEST.COM
I20260928 04:47:36.397938 24281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.390556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34842 (local address 127.1.226.253:35891)
0928 04:47:36.390596 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:36.390598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:36.390608 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:47:36.390684 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:36.390685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:36.390697 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:36.390770 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:36.391102 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.391645 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.392447 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.392683 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.392759 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:36.394887 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:36.394922 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:36.394922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:36.394929 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:36.396679 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.397136 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.397138 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.397139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.397201 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.397399 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.397400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.397400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.397526 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:36.397604 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:36.397757 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 04:47:36.397839 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:47:36.397945 24204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.390507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:35891 (local address 127.0.0.1:34842)
0928 04:47:36.390555 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:36.390565 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:36.390617 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:36.390777 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:36.390777 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:36.390948 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:36.391013 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:36.391015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.391745 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.391746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:36.392361 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:36.392364 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.392769 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.392770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:36.392855 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:36.393343 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:47:36.393348 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:36.394795 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:36.396867 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:36.396868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:36.396869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:36.397050 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:36.397269 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:36.397270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:36.397270 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:36.397333 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:36.397595 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:36.397596 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:36.397652 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 04:47:36.397732 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 04:47:36.397852 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 04:47:36.398590 24202 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.252" port: 37119 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:47:36.398985 24204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.398847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:37119 (local address 127.0.0.1:34352)
0928 04:47:36.398924 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:36.398954 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 04:47:36.399362 24204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.399276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:37119 (local address 127.0.0.1:34362)
0928 04:47:36.399316 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:36.399340 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 04:47:36.399479 24202 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.226.252:37119: Network error: Client connection negotiation failed: client connection to 127.1.226.252:37119: connect: Connection refused (error 111)
I20260928 04:47:36.430699 24202 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.226.252" port: 37119 } attempt: 1
I20260928 04:47:36.431272 24204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.431111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:37119 (local address 127.0.0.1:34370)
0928 04:47:36.431176 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:36.431236 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 04:47:36.431650 24204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.431554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:37119 (local address 127.0.0.1:34384)
0928 04:47:36.431599 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:36.431629 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 04:47:36.431782 24202 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.226.252:37119: Network error: Client connection negotiation failed: client connection to 127.1.226.252:37119: connect: Connection refused (error 111)
W20260928 04:47:36.432054 24284 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 04:47:36.432327 24284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:36.432379 24284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:36.432420 24284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:36.433599 24284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:36.433640 24284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:36.433655 24284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:36.433713 24284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:36.433734 24284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:36.435286 24284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:42185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:45325,127.1.226.253:35891,127.1.226.252:37119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.252.keytab
--principal=kudu/127.1.226.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.252:37119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.226.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:36.435575 24284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:36.435854 24284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:36.440964 24292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:36.440969 24290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:36.440980 24289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:36.441417 24284 server_base.cc:1034] running on GCE node
I20260928 04:47:36.441677 24284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:36.442010 24284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:36.443189 24284 hybrid_clock.cc:648] HybridClock initialized: now 1790570856443164 us; error 34 us; skew 500 ppm
Sep 28 04:47:36 dist-test-slave-j5c3 krb5kdc[24121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570856, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:36.446050 24284 init.cc:377] Logged in from keytab as kudu/127.1.226.252@KRBTEST.COM (short username kudu)
I20260928 04:47:36.446519 24284 webserver.cc:460] Webserver started at http://127.1.226.252:43669/ using document root <none> and password file <none>
I20260928 04:47:36.446764 24284 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:36.446842 24284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:36.446992 24284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:36.448076 24284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-2/data/instance:
uuid: "401a24d6c6b04fbba2c30b7d5298df91"
format_stamp: "Formatted at 2026-09-28 04:47:36 on dist-test-slave-j5c3"
server_key: "4985f519f85727f207123f36468ba1bc"
server_key_iv: "3907ec32825a47cdfa1d62bff0164588"
server_key_version: "encryptionkey@0"
I20260928 04:47:36.448513 24284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-2/wal/instance:
uuid: "401a24d6c6b04fbba2c30b7d5298df91"
format_stamp: "Formatted at 2026-09-28 04:47:36 on dist-test-slave-j5c3"
server_key: "4985f519f85727f207123f36468ba1bc"
server_key_iv: "3907ec32825a47cdfa1d62bff0164588"
server_key_version: "encryptionkey@0"
I20260928 04:47:36.450026 24284 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:36.450842 24299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:36.451051 24284 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:36.451152 24284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-2/wal
uuid: "401a24d6c6b04fbba2c30b7d5298df91"
format_stamp: "Formatted at 2026-09-28 04:47:36 on dist-test-slave-j5c3"
server_key: "4985f519f85727f207123f36468ba1bc"
server_key_iv: "3907ec32825a47cdfa1d62bff0164588"
server_key_version: "encryptionkey@0"
I20260928 04:47:36.451256 24284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:36.458433 24277 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.226.252" port: 37119 } attempt: 2
I20260928 04:47:36.458927 24279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.458772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:37119 (local address 127.0.0.1:34400)
0928 04:47:36.458838 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:36.458888 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 04:47:36.459299 24279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.459211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:37119 (local address 127.0.0.1:34408)
0928 04:47:36.459255 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:36.459281 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 04:47:36.459506 24277 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.226.252:37119: Network error: Client connection negotiation failed: client connection to 127.1.226.252:37119: connect: Connection refused (error 111)
I20260928 04:47:36.471949 24284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:36.474385 24284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:36.474584 24284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:36.479020 24284 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.252:37119
I20260928 04:47:36.479063 24351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.252:37119 every 8 connection(s)
I20260928 04:47:36.479452 24284 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-2/data/info.pb
I20260928 04:47:36.480621 24352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:36.481788 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 24284
I20260928 04:47:36.481927 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570585693787-1931-0/minicluster-data/master-2/wal/instance
I20260928 04:47:36.482128 1931 external_mini_cluster.cc:1428] Setting key 63afdf33d27d0dd82d38151c6ca18b96
I20260928 04:47:36.482436 24352 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45325 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 04:47:36 dist-test-slave-j5c3 krb5kdc[24121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570856, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.252@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
Sep 28 04:47:36 dist-test-slave-j5c3 krb5kdc[24121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.252@KRBTEST.COM
I20260928 04:47:36.492097 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.483140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37316 (local address 127.1.226.254:45325)
0928 04:47:36.483194 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:36.483196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:36.483315 (+ 119us) server_negotiation.cc:407] Connection header received
0928 04:47:36.483652 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:36.483653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:36.483664 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:36.483724 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:36.484404 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.484930 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.486086 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.486399 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.486477 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:36.489069 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:36.489071 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:36.489072 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:36.489080 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:36.490602 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.491032 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.491033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.491034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.491088 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.491412 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.491413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.491413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.491526 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:36.491601 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:36.491903 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 04:47:36.491994 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 04:47:36.492334 24356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.483074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:45325 (local address 127.0.0.1:37316)
0928 04:47:36.483185 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:36.483224 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:36.483550 (+ 326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:36.483743 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:36.483745 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:36.484057 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:36.484298 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:36.484303 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.485010 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.485011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:36.486000 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:36.486002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.486526 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.486527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:36.486664 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:36.487262 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:47:36.487271 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:36.488967 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:36.490683 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:36.490691 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:36.490694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:36.490948 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:36.491179 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:36.491180 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:36.491181 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:36.491310 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:36.491606 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:36.491614 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:36.491764 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 04:47:36.491880 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 04:47:36.492052 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":41,"thread_start_us":58,"threads_started":1}
I20260928 04:47:36.492976 24355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.482724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34418 (local address 127.1.226.252:37119)
0928 04:47:36.482914 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:36.482924 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:36.482958 (+ 34us) server_negotiation.cc:407] Connection header received
0928 04:47:36.483503 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:36.483511 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:36.483533 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:36.483653 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:36.484522 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.485600 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.486545 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.486969 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.487037 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:36.489708 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:36.489716 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:36.489716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:36.489722 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:36.491585 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.491960 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.491963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.491964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.492021 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.492304 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.492305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.492306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.492493 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:36.492600 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:36.492770 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 04:47:36.492854 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":82,"threads_started":1}
I20260928 04:47:36.493165 24352 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.253" port: 35891 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 04:47:36 dist-test-slave-j5c3 krb5kdc[24121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570856, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.252@KRBTEST.COM for kudu/127.1.226.253@KRBTEST.COM
Sep 28 04:47:36 dist-test-slave-j5c3 krb5kdc[24121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:47:36.501060 24281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.493521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34858 (local address 127.1.226.253:35891)
0928 04:47:36.493575 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:36.493577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:36.493600 (+ 23us) server_negotiation.cc:407] Connection header received
0928 04:47:36.493628 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:36.493629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:36.493641 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:36.493702 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:36.494126 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.494672 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.495465 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.495666 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.495719 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:36.498018 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:36.498020 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:36.498020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:36.498029 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:36.499835 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.500270 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.500272 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.500272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.500337 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.500544 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.500545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.500546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.500696 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:36.500771 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:36.500919 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 04:47:36.500993 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:47:36.501099 24356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.493438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:35891 (local address 127.0.0.1:34858)
0928 04:47:36.493489 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:36.493502 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:36.493542 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:36.493720 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:36.493721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:36.493979 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:36.494047 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:36.494049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.494761 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.494762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:36.495360 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:36.495362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.495736 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.495737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:36.495809 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:36.496413 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:47:36.496420 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:36.497893 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:36.499925 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:36.499926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:36.499927 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:36.500145 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:36.500423 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:36.500424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:36.500424 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:36.500480 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:36.500764 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:36.500764 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:36.500801 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 04:47:36.500874 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 04:47:36.500988 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 04:47:36.501509 24352 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.252" port: 37119 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:47:36.502096 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.494015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37330 (local address 127.1.226.254:45325)
0928 04:47:36.494066 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:36.494068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:36.494077 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:47:36.494117 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:36.494118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:36.494127 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:36.494176 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:36.494638 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.495127 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.496032 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.496368 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.496433 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:36.499076 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:36.499078 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:36.499078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:36.499087 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:36.500816 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.501219 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.501221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.501221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.501281 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.501507 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.501508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.501508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.501680 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:36.501877 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:36.501918 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 04:47:36.501990 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Sep 28 04:47:36 dist-test-slave-j5c3 krb5kdc[24121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570856, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.252@KRBTEST.COM for kudu/127.1.226.252@KRBTEST.COM
Sep 28 04:47:36 dist-test-slave-j5c3 krb5kdc[24121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.253@KRBTEST.COM
I20260928 04:47:36.507191 24202 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.226.252" port: 37119 } attempt: 2
I20260928 04:47:36.509173 24356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.501789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:37119 (local address 127.0.0.1:34434)
0928 04:47:36.501846 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:36.501858 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:36.501894 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:36.502070 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:36.502071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:36.502349 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:36.502390 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:36.502393 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.503202 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.503203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:36.503817 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:36.503819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.504373 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.504374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:36.504428 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:36.504816 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:47:36.504821 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:36.506214 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:36.508212 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:36.508213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:36.508213 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:36.508464 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:36.508649 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:36.508649 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:36.508650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:36.508691 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:36.508901 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:36.508902 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:36.508932 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 04:47:36.508971 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 04:47:36.509061 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 04:47:36.509573 24355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.501860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34434 (local address 127.1.226.252:37119)
0928 04:47:36.501956 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:36.501958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:36.501964 (+ 6us) server_negotiation.cc:407] Connection header received
0928 04:47:36.501980 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:36.501981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:36.501990 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:36.502045 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:36.502528 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.503123 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.503907 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.504285 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.504339 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:36.506309 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:36.506310 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:36.506310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:36.506317 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:36.507787 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.508544 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.508545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.508545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.508591 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.508745 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.508745 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.508745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.508838 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:36.509379 (+ 541us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:36.509429 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 04:47:36.509497 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 04:47:36.510723 24281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.502774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34868 (local address 127.1.226.253:35891)
0928 04:47:36.502820 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:36.502822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:36.502835 (+ 13us) server_negotiation.cc:407] Connection header received
0928 04:47:36.502857 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:36.502858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:36.502867 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:36.502922 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:36.503483 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.504035 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.504992 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.505184 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.505242 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:36.507744 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:36.507746 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:36.507746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:36.507753 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:36.509347 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.509745 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.509746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.509746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.509799 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.510015 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.510017 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.510017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.510161 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:36.510319 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:36.510537 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 04:47:36.510620 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:47:36.511487 24352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401a24d6c6b04fbba2c30b7d5298df91: Bootstrap starting.
I20260928 04:47:36.512187 24352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 401a24d6c6b04fbba2c30b7d5298df91: Neither blocks nor log segments found. Creating new log.
Sep 28 04:47:36 dist-test-slave-j5c3 krb5kdc[24121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570856, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for kudu/127.1.226.252@KRBTEST.COM
I20260928 04:47:36.512562 24352 log.cc:824] T 00000000000000000000000000000000 P 401a24d6c6b04fbba2c30b7d5298df91: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:36.513439 24352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401a24d6c6b04fbba2c30b7d5298df91: No bootstrap required, opened a new log
I20260928 04:47:36.513845 1931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:47:36.515241 24352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 401a24d6c6b04fbba2c30b7d5298df91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c813bd87b44d9848cb7a2fcc81e44" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45325 } } peers { permanent_uuid: "5da39d55d9b94877ac32dddea369a852" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 35891 } } peers { permanent_uuid: "401a24d6c6b04fbba2c30b7d5298df91" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 37119 } }
I20260928 04:47:36.515409 24352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 401a24d6c6b04fbba2c30b7d5298df91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401a24d6c6b04fbba2c30b7d5298df91, State: Initialized, Role: FOLLOWER
I20260928 04:47:36.515491 24352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 401a24d6c6b04fbba2c30b7d5298df91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c813bd87b44d9848cb7a2fcc81e44" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45325 } } peers { permanent_uuid: "5da39d55d9b94877ac32dddea369a852" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 35891 } } peers { permanent_uuid: "401a24d6c6b04fbba2c30b7d5298df91" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 37119 } }
I20260928 04:47:36.514786 24176 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "573c813bd87b44d9848cb7a2fcc81e44"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37330
I20260928 04:47:36.515817 24352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 401a24d6c6b04fbba2c30b7d5298df91 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:36.515756 24359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401a24d6c6b04fbba2c30b7d5298df91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c813bd87b44d9848cb7a2fcc81e44" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45325 } } peers { permanent_uuid: "5da39d55d9b94877ac32dddea369a852" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 35891 } } peers { permanent_uuid: "401a24d6c6b04fbba2c30b7d5298df91" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 37119 } } }
I20260928 04:47:36.515908 24359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401a24d6c6b04fbba2c30b7d5298df91 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 04:47:36.517827 24370 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 401a24d6c6b04fbba2c30b7d5298df91: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:47:36.518324 24370 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:47:36.520033 24204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.507463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:37119 (local address 127.0.0.1:34440)
0928 04:47:36.507520 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:36.507530 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:36.507567 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:36.507841 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:36.507843 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:36.508035 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:36.508087 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:36.508090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.509119 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.509120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:36.510464 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:36.510467 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.510887 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.510900 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:36.510959 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:36.511359 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:47:36.511363 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:36.512791 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:36.518864 (+ 6073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:36.518865 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:36.518866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:36.519108 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:36.519341 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:36.519342 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:36.519342 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:36.519411 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:36.519692 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:36.519693 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:36.519744 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 04:47:36.519834 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 04:47:36.519941 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 04:47:36.520047 24357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.507515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34440 (local address 127.1.226.252:37119)
0928 04:47:36.507704 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:36.507706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:36.507713 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:47:36.507733 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:36.507734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:36.507746 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:36.507835 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:36.508187 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.509025 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.510557 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.510788 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.511126 (+ 338us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:36.512888 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:36.512889 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:36.512890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:36.512896 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:36.518773 (+ 5877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.519196 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.519198 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.519198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.519260 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.519478 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.519479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.519480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.519620 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:36.519699 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:36.519859 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 04:47:36.519943 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":122,"threads_started":1}
I20260928 04:47:36.521364 24202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 573c813bd87b44d9848cb7a2fcc81e44: Bootstrap starting.
I20260928 04:47:36.519217 24284 master_runner.cc:186] Error getting master registration for 127.1.226.254:45325: OK, instance_id { permanent_uuid: "573c813bd87b44d9848cb7a2fcc81e44" instance_seqno: 1790570856160671 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 04:47:36.521510 24284 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 04:47:36.522028 24202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 573c813bd87b44d9848cb7a2fcc81e44: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:36.522329 24202 log.cc:824] T 00000000000000000000000000000000 P 573c813bd87b44d9848cb7a2fcc81e44: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:36.523042 24202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 573c813bd87b44d9848cb7a2fcc81e44: No bootstrap required, opened a new log
I20260928 04:47:36.524013 24202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 573c813bd87b44d9848cb7a2fcc81e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c813bd87b44d9848cb7a2fcc81e44" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45325 } } peers { permanent_uuid: "5da39d55d9b94877ac32dddea369a852" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 35891 } } peers { permanent_uuid: "401a24d6c6b04fbba2c30b7d5298df91" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 37119 } }
I20260928 04:47:36.524143 24202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 573c813bd87b44d9848cb7a2fcc81e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573c813bd87b44d9848cb7a2fcc81e44, State: Initialized, Role: FOLLOWER
I20260928 04:47:36.524271 24202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 573c813bd87b44d9848cb7a2fcc81e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c813bd87b44d9848cb7a2fcc81e44" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45325 } } peers { permanent_uuid: "5da39d55d9b94877ac32dddea369a852" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 35891 } } peers { permanent_uuid: "401a24d6c6b04fbba2c30b7d5298df91" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 37119 } }
I20260928 04:47:36.524571 24373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573c813bd87b44d9848cb7a2fcc81e44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c813bd87b44d9848cb7a2fcc81e44" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45325 } } peers { permanent_uuid: "5da39d55d9b94877ac32dddea369a852" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 35891 } } peers { permanent_uuid: "401a24d6c6b04fbba2c30b7d5298df91" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 37119 } } }
I20260928 04:47:36.524636 24202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 573c813bd87b44d9848cb7a2fcc81e44 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:36.524649 24373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573c813bd87b44d9848cb7a2fcc81e44 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:47:36.524736 24176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 573c813bd87b44d9848cb7a2fcc81e44 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 04:47:36.524811 24176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 573c813bd87b44d9848cb7a2fcc81e44 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:36.525673 24176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 573c813bd87b44d9848cb7a2fcc81e44 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c813bd87b44d9848cb7a2fcc81e44" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45325 } } peers { permanent_uuid: "5da39d55d9b94877ac32dddea369a852" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 35891 } } peers { permanent_uuid: "401a24d6c6b04fbba2c30b7d5298df91" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 37119 } }
I20260928 04:47:36.526166 24176 leader_election.cc:288] T 00000000000000000000000000000000 P 573c813bd87b44d9848cb7a2fcc81e44 [CANDIDATE]: Term 1 election: Requested vote from peers 5da39d55d9b94877ac32dddea369a852 (127.1.226.253:35891), 401a24d6c6b04fbba2c30b7d5298df91 (127.1.226.252:37119)
I20260928 04:47:36.526577 24326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "573c813bd87b44d9848cb7a2fcc81e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "401a24d6c6b04fbba2c30b7d5298df91"
I20260928 04:47:36.526733 24326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 401a24d6c6b04fbba2c30b7d5298df91 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:36.527704 24326 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 401a24d6c6b04fbba2c30b7d5298df91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 573c813bd87b44d9848cb7a2fcc81e44 in term 1.
I20260928 04:47:36.527971 24150 leader_election.cc:302] T 00000000000000000000000000000000 P 573c813bd87b44d9848cb7a2fcc81e44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401a24d6c6b04fbba2c30b7d5298df91, 573c813bd87b44d9848cb7a2fcc81e44; no voters:
I20260928 04:47:36.528113 24373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 573c813bd87b44d9848cb7a2fcc81e44 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:36.528298 24373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 573c813bd87b44d9848cb7a2fcc81e44 [term 1 LEADER]: Becoming Leader. State: Replica: 573c813bd87b44d9848cb7a2fcc81e44, State: Running, Role: LEADER
I20260928 04:47:36.528388 24373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 573c813bd87b44d9848cb7a2fcc81e44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c813bd87b44d9848cb7a2fcc81e44" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45325 } } peers { permanent_uuid: "5da39d55d9b94877ac32dddea369a852" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 35891 } } peers { permanent_uuid: "401a24d6c6b04fbba2c30b7d5298df91" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 37119 } }
W20260928 04:47:36.529652 24384 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 573c813bd87b44d9848cb7a2fcc81e44: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:47:36.529708 24384 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:47:36.528944 24389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573c813bd87b44d9848cb7a2fcc81e44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 573c813bd87b44d9848cb7a2fcc81e44. Latest consensus state: current_term: 1 leader_uuid: "573c813bd87b44d9848cb7a2fcc81e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c813bd87b44d9848cb7a2fcc81e44" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45325 } } peers { permanent_uuid: "5da39d55d9b94877ac32dddea369a852" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 35891 } } peers { permanent_uuid: "401a24d6c6b04fbba2c30b7d5298df91" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 37119 } } }
I20260928 04:47:36.529775 24389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573c813bd87b44d9848cb7a2fcc81e44 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:36.529954 24392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:47:36.530112 24392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:47:36.527180 24134 master_runner.cc:186] Error getting master registration for 127.1.226.253:35891: OK, instance_id { permanent_uuid: "5da39d55d9b94877ac32dddea369a852" instance_seqno: 1790570856313055 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 04:47:36.530241 24134 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 04:47:36.531821 24326 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 401a24d6c6b04fbba2c30b7d5298df91 [term 1 FOLLOWER]: Refusing update from remote peer 573c813bd87b44d9848cb7a2fcc81e44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:47:36.532270 24389 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 573c813bd87b44d9848cb7a2fcc81e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "401a24d6c6b04fbba2c30b7d5298df91" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 37119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:47:36.533525 24359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401a24d6c6b04fbba2c30b7d5298df91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 573c813bd87b44d9848cb7a2fcc81e44. Latest consensus state: current_term: 1 leader_uuid: "573c813bd87b44d9848cb7a2fcc81e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c813bd87b44d9848cb7a2fcc81e44" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45325 } } peers { permanent_uuid: "5da39d55d9b94877ac32dddea369a852" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 35891 } } peers { permanent_uuid: "401a24d6c6b04fbba2c30b7d5298df91" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 37119 } } }
I20260928 04:47:36.533615 24359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401a24d6c6b04fbba2c30b7d5298df91 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:47:36.545154 24357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.530566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34444 (local address 127.1.226.252:37119)
0928 04:47:36.530608 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:36.530610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:36.533777 (+ 3167us) server_negotiation.cc:407] Connection header received
0928 04:47:36.533867 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:36.533868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:36.533881 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:36.533932 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:36.534865 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.535648 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.537030 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.537354 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.537423 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:36.540352 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:36.540354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:36.540354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:36.540362 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:36.542044 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.542442 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.542443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.542444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.542499 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.542884 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.542885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.542886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.544119 (+ 1233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:36.544277 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:36.544936 (+ 659us) server_negotiation.cc:299] Negotiation successful
0928 04:47:36.545026 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:47:36.545943 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.530776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37342 (local address 127.1.226.254:45325)
0928 04:47:36.530819 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:36.530821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:36.532614 (+ 1793us) server_negotiation.cc:407] Connection header received
0928 04:47:36.532714 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:36.532715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:36.532726 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:36.532755 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:36.534884 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.535622 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.537530 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.537818 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.537876 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:36.539387 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:36.539395 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:36.539396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:36.539404 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:36.541339 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.542694 (+ 1355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.542695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.542696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.542747 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.544589 (+ 1842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.544591 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.544591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.544724 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:36.544803 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:36.545717 (+ 914us) server_negotiation.cc:299] Negotiation successful
0928 04:47:36.545807 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:47:36.547765 24389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573c813bd87b44d9848cb7a2fcc81e44 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "573c813bd87b44d9848cb7a2fcc81e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c813bd87b44d9848cb7a2fcc81e44" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45325 } } peers { permanent_uuid: "5da39d55d9b94877ac32dddea369a852" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 35891 } } peers { permanent_uuid: "401a24d6c6b04fbba2c30b7d5298df91" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 37119 } } }
I20260928 04:47:36.547871 24389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573c813bd87b44d9848cb7a2fcc81e44 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:36.547969 24393 mvcc.cc:205] Tried to move back new op lower bound from 7334178228351533056 to 7334178228340977664. Current Snapshot: MvccSnapshot[applied={T|T < 7334178228351533056}]
I20260928 04:47:36.548568 24359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401a24d6c6b04fbba2c30b7d5298df91 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "573c813bd87b44d9848cb7a2fcc81e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c813bd87b44d9848cb7a2fcc81e44" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45325 } } peers { permanent_uuid: "5da39d55d9b94877ac32dddea369a852" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 35891 } } peers { permanent_uuid: "401a24d6c6b04fbba2c30b7d5298df91" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 37119 } } }
I20260928 04:47:36.548668 24359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401a24d6c6b04fbba2c30b7d5298df91 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:47:36.548944 24397 mvcc.cc:205] Tried to move back new op lower bound from 7334178228351533056 to 7334178228340977664. Current Snapshot: MvccSnapshot[applied={T|T < 7334178228351533056}]
I20260928 04:47:36.550685 24277 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.226.252" port: 37119 } attempt: 3
I20260928 04:47:36.552122 24392 catalog_manager.cc:1347] Generated new cluster ID: 7e6468cfa19a4b2a9733d88c3f972bcd
I20260928 04:47:36.552192 24392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:47:36 dist-test-slave-j5c3 krb5kdc[24121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570856, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.253@KRBTEST.COM for kudu/127.1.226.252@KRBTEST.COM
I20260928 04:47:36.564667 24279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.550952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:37119 (local address 127.0.0.1:34460)
0928 04:47:36.551019 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:36.551029 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:36.551069 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:36.551210 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:36.551211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:36.551377 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:36.551429 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:36.551431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.556849 (+ 5418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.556851 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:36.557650 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:36.557652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.559098 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.559099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:36.559172 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:36.559599 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:47:36.559617 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:36.561095 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:36.563451 (+ 2356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:36.563453 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:36.563454 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:36.563684 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:36.563880 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:36.563880 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:36.563881 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:36.563937 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:36.564361 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:36.564362 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:36.564417 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 04:47:36.564460 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 04:47:36.564562 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 04:47:36.565605 24357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.551029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34460 (local address 127.1.226.252:37119)
0928 04:47:36.551085 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:36.551086 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:36.551094 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:47:36.551131 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:36.551132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:36.551143 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:36.551204 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:36.551490 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.552533 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.558650 (+ 6117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.558975 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.559029 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:36.561227 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:36.561228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:36.561229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:36.561253 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:36.563335 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.563750 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.563752 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.563752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.563810 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.564003 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.564004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.564004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.564278 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:36.565262 (+ 984us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:36.565332 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 04:47:36.565462 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 04:47:36.567373 24277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5da39d55d9b94877ac32dddea369a852: Bootstrap starting.
I20260928 04:47:36.568077 24277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5da39d55d9b94877ac32dddea369a852: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:36.568377 24277 log.cc:824] T 00000000000000000000000000000000 P 5da39d55d9b94877ac32dddea369a852: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:36.569542 24277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5da39d55d9b94877ac32dddea369a852: No bootstrap required, opened a new log
I20260928 04:47:36.571427 24277 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5da39d55d9b94877ac32dddea369a852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c813bd87b44d9848cb7a2fcc81e44" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45325 } } peers { permanent_uuid: "5da39d55d9b94877ac32dddea369a852" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 35891 } } peers { permanent_uuid: "401a24d6c6b04fbba2c30b7d5298df91" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 37119 } }
I20260928 04:47:36.571568 24277 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5da39d55d9b94877ac32dddea369a852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da39d55d9b94877ac32dddea369a852, State: Initialized, Role: FOLLOWER
I20260928 04:47:36.571636 24277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5da39d55d9b94877ac32dddea369a852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c813bd87b44d9848cb7a2fcc81e44" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45325 } } peers { permanent_uuid: "5da39d55d9b94877ac32dddea369a852" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 35891 } } peers { permanent_uuid: "401a24d6c6b04fbba2c30b7d5298df91" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 37119 } }
I20260928 04:47:36.571988 24277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5da39d55d9b94877ac32dddea369a852 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:36.572203 24281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:36.531081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34880 (local address 127.1.226.253:35891)
0928 04:47:36.531122 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:36.531124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:36.538305 (+ 7181us) server_negotiation.cc:407] Connection header received
0928 04:47:36.542860 (+ 4555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:36.542873 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:36.542890 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:36.542941 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:36.544306 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.545040 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.548913 (+ 3873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:36.549232 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:36.549284 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:36.551776 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:36.551778 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:36.551778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:36.551785 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:36.558536 (+ 6751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.561252 (+ 2716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.561254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.561254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.561325 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:36.561823 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:36.561824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:36.561824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:36.561969 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:36.562067 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:36.571968 (+ 9901us) server_negotiation.cc:299] Negotiation successful
0928 04:47:36.572088 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:47:36.572535 24250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5da39d55d9b94877ac32dddea369a852 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:36.573496 24250 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5da39d55d9b94877ac32dddea369a852 [term 1 FOLLOWER]: Refusing update from remote peer 573c813bd87b44d9848cb7a2fcc81e44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:47:36.573571 24402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5da39d55d9b94877ac32dddea369a852 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c813bd87b44d9848cb7a2fcc81e44" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45325 } } peers { permanent_uuid: "5da39d55d9b94877ac32dddea369a852" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 35891 } } peers { permanent_uuid: "401a24d6c6b04fbba2c30b7d5298df91" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 37119 } } }
I20260928 04:47:36.573637 24402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5da39d55d9b94877ac32dddea369a852 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:47:36.573684 24251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "573c813bd87b44d9848cb7a2fcc81e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5da39d55d9b94877ac32dddea369a852"
I20260928 04:47:36.574393 24251 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5da39d55d9b94877ac32dddea369a852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 573c813bd87b44d9848cb7a2fcc81e44 in term 1.
I20260928 04:47:36.575008 24373 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 573c813bd87b44d9848cb7a2fcc81e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5da39d55d9b94877ac32dddea369a852" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 35891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:47:36.576342 24402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5da39d55d9b94877ac32dddea369a852 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 573c813bd87b44d9848cb7a2fcc81e44. Latest consensus state: current_term: 1 leader_uuid: "573c813bd87b44d9848cb7a2fcc81e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c813bd87b44d9848cb7a2fcc81e44" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45325 } } peers { permanent_uuid: "5da39d55d9b94877ac32dddea369a852" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 35891 } } peers { permanent_uuid: "401a24d6c6b04fbba2c30b7d5298df91" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 37119 } } }
I20260928 04:47:36.576463 24402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5da39d55d9b94877ac32dddea369a852 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:47:36.576507 24402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5da39d55d9b94877ac32dddea369a852 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "573c813bd87b44d9848cb7a2fcc81e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c813bd87b44d9848cb7a2fcc81e44" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45325 } } peers { permanent_uuid: "5da39d55d9b94877ac32dddea369a852" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 35891 } } peers { permanent_uuid: "401a24d6c6b04fbba2c30b7d5298df91" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 37119 } } }
I20260928 04:47:36.576568 24402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5da39d55d9b94877ac32dddea369a852 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:47:36.585539 24403 mvcc.cc:205] Tried to move back new op lower bound from 7334178228351533056 to 7334178228340977664. Current Snapshot: MvccSnapshot[applied={T|T < 7334178228351533056}]
W20260928 04:47:36.586354 24416 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5da39d55d9b94877ac32dddea369a852: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:47:36.586457 24416 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:47:36.612474 24373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573c813bd87b44d9848cb7a2fcc81e44 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "573c813bd87b44d9848cb7a2fcc81e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c813bd87b44d9848cb7a2fcc81e44" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 45325 } } peers { permanent_uuid: "5da39d55d9b94877ac32dddea369a852" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 35891 } } peers { permanent_uuid: "401a24d6c6b04fbba2c30b7d5298df91" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 37119 } } }
I20260928 04:47:36.612612 24373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573c813bd87b44d9848cb7a2fcc81e44 [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:36.614291 24392 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:47:36.615061 24392 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:47:36.619825 24392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 573c813bd87b44d9848cb7a2fcc81e44: Generated new TSK 0
I20260928 04:47:36.620026 24392 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 04:47:36.641999 24205 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:47:36.640451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37354 (local address 127.1.226.254:45325)
0928 04:47:36.640492 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:36.640494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:36.641274 (+ 780us) server_negotiation.cc:407] Connection header received
0928 04:47:36.641385 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:36.641386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:36.641398 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:36.641430 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:36.641919 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37354: BlockingRecv error: recv got EOF from 127.0.0.1:37354 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 04:47:36.642979 24357 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:47:36.640225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34476 (local address 127.1.226.252:37119)
0928 04:47:36.640276 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:36.640279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:36.642392 (+ 2113us) server_negotiation.cc:407] Connection header received
0928 04:47:36.642493 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:36.642494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:36.642507 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:36.642537 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:36.642910 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34476: BlockingRecv error: recv got EOF from 127.0.0.1:34476 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 04:47:36.643841 24281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:47:36.640882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34888 (local address 127.1.226.253:35891)
0928 04:47:36.640936 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:36.640938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:36.643342 (+ 2404us) server_negotiation.cc:407] Connection header received
0928 04:47:36.643441 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:36.643442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:36.643454 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:36.643483 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:36.643777 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34888: BlockingRecv error: recv got EOF from 127.0.0.1:34888 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 04:47:36.646675 24281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:47:36.645804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34890 (local address 127.1.226.253:35891)
0928 04:47:36.645841 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:36.645843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:36.646074 (+ 231us) server_negotiation.cc:407] Connection header received
0928 04:47:36.646183 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:36.646184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:36.646203 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:36.646232 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:36.646626 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34890: BlockingRecv error: recv got EOF from 127.0.0.1:34890 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 04:47:36.653723 24281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:47:36.652653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34892 (local address 127.1.226.253:35891)
0928 04:47:36.652712 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:36.652714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:36.653065 (+ 351us) server_negotiation.cc:407] Connection header received
0928 04:47:36.653180 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:36.653181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:36.653194 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:36.653229 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:36.653640 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34892: BlockingRecv error: recv got EOF from 127.0.0.1:34892 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 04:47:36.658146 24281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:47:36.657251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34904 (local address 127.1.226.253:35891)
0928 04:47:36.657293 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:36.657295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:36.657544 (+ 249us) server_negotiation.cc:407] Connection header received
0928 04:47:36.657670 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:36.657671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:36.657683 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:36.657726 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:36.658088 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34904: BlockingRecv error: recv got EOF from 127.0.0.1:34904 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 04:47:36.664716 24281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:47:36.663614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34918 (local address 127.1.226.253:35891)
0928 04:47:36.663656 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:36.663658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:36.663972 (+ 314us) server_negotiation.cc:407] Connection header received
0928 04:47:36.664083 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:36.664085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:36.664096 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:36.664130 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:36.664644 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34918: BlockingRecv error: recv got EOF from 127.0.0.1:34918 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 04:47:36.675318 24281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:47:36.674324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34920 (local address 127.1.226.253:35891)
0928 04:47:36.674371 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:36.674374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:36.674667 (+ 293us) server_negotiation.cc:407] Connection header received
0928 04:47:36.674799 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:36.674800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:36.674813 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:36.674848 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:36.675250 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34920: BlockingRecv error: recv got EOF from 127.0.0.1:34920 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 04:47:36.694036 24281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:47:36.693008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34926 (local address 127.1.226.253:35891)
0928 04:47:36.693055 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:36.693058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:36.693363 (+ 305us) server_negotiation.cc:407] Connection header received
0928 04:47:36.693485 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:36.693486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:36.693500 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:36.693534 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:36.693972 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34926: BlockingRecv error: recv got EOF from 127.0.0.1:34926 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 04:47:36.728555 24281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:47:36.727741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34938 (local address 127.1.226.253:35891)
0928 04:47:36.727819 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:36.727822 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:36.727911 (+ 89us) server_negotiation.cc:407] Connection header received
0928 04:47:36.727955 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:36.727957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:36.727974 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:36.728082 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:36.728485 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34938: BlockingRecv error: recv got EOF from 127.0.0.1:34938 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 04:47:36.795042 24281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:47:36.794136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34952 (local address 127.1.226.253:35891)
0928 04:47:36.794208 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:36.794211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:36.794355 (+ 144us) server_negotiation.cc:407] Connection header received
0928 04:47:36.794412 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:36.794413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:36.794425 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:36.794482 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:36.794960 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34952: BlockingRecv error: recv got EOF from 127.0.0.1:34952 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 04:47:36.925243 24281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:47:36.924262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34966 (local address 127.1.226.253:35891)
0928 04:47:36.924337 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:36.924339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:36.924579 (+ 240us) server_negotiation.cc:407] Connection header received
0928 04:47:36.924681 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:36.924682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:36.924694 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:36.924771 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:36.925194 (+ 423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34966: BlockingRecv error: recv got EOF from 127.0.0.1:34966 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 04:47:37.183365 24281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:47:37.182679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34974 (local address 127.1.226.253:35891)
0928 04:47:37.182761 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:37.182764 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:37.182775 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:47:37.182856 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:37.182858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:37.182870 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:37.182949 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:37.183318 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34974: BlockingRecv error: recv got EOF from 127.0.0.1:34974 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 04:47:37.518833 24370 catalog_manager.cc:1259] Loaded cluster ID: 7e6468cfa19a4b2a9733d88c3f972bcd
I20260928 04:47:37.518913 24370 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 401a24d6c6b04fbba2c30b7d5298df91: loading cluster ID for follower catalog manager: success
I20260928 04:47:37.519869 24370 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 401a24d6c6b04fbba2c30b7d5298df91: acquiring CA information for follower catalog manager: success
I20260928 04:47:37.520255 24370 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 401a24d6c6b04fbba2c30b7d5298df91: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 04:47:37.586992 24416 catalog_manager.cc:1259] Loaded cluster ID: 7e6468cfa19a4b2a9733d88c3f972bcd
I20260928 04:47:37.587075 24416 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5da39d55d9b94877ac32dddea369a852: loading cluster ID for follower catalog manager: success
I20260928 04:47:37.587925 24416 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5da39d55d9b94877ac32dddea369a852: acquiring CA information for follower catalog manager: success
I20260928 04:47:37.588307 24416 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5da39d55d9b94877ac32dddea369a852: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 04:47:37.699795 24483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:37.696922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34990 (local address 127.1.226.253:35891)
0928 04:47:37.697058 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:37.697060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:37.697088 (+ 28us) server_negotiation.cc:407] Connection header received
0928 04:47:37.697208 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:37.697214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:37.697307 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:37.697388 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:47:37.697555 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:37.698066 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:37.699062 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:37.699264 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:37.699356 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:37.699495 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:47:37.699606 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:47:37.699666 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:37.699707 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 04:47:37.699726 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":54,"threads_started":1}
I20260928 04:47:37.704278 24489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:37.701938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34480 (local address 127.1.226.252:37119)
0928 04:47:37.702040 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:37.702042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:37.702049 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:47:37.702084 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:37.702085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:37.702121 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:37.702202 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:47:37.702304 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:37.702795 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:37.703527 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:37.703720 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:37.703793 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:37.703920 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:47:37.704023 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:47:37.704080 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:37.704110 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 04:47:37.704127 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":58,"threads_started":1}
I20260928 04:47:37.709267 24483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:37.706137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34994 (local address 127.1.226.253:35891)
0928 04:47:37.706290 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:37.706292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:37.706517 (+ 225us) server_negotiation.cc:407] Connection header received
0928 04:47:37.706573 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:37.706574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:37.706585 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:37.706653 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:47:37.706803 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:37.707273 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:37.708346 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:37.708657 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:37.708752 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:37.708945 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:47:37.708987 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:47:37.709022 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:37.709130 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 04:47:37.709156 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260928 04:47:37.709342 24489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:37.706339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34484 (local address 127.1.226.252:37119)
0928 04:47:37.706397 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:37.706399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:37.706407 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:47:37.706438 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:37.706439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:37.706450 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:37.706507 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:47:37.706696 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:37.707522 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:37.708536 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:37.708811 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:37.708874 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:37.709060 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:47:37.709095 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:47:37.709143 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:37.709231 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 04:47:37.709252 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 04:47:37.709997 24496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:37.706050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37370 (local address 127.1.226.254:45325)
0928 04:47:37.706173 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:37.706175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:37.706185 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:47:37.706314 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:37.706315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:37.706373 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:37.706454 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:47:37.706612 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:37.707536 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:37.708573 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:37.708886 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:37.708957 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:37.709154 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:47:37.709269 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:47:37.709330 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:37.709738 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 04:47:37.709782 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":68,"threads_started":1}
I20260928 04:47:37.711743 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 24134
I20260928 04:47:37.717589 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 24209
I20260928 04:47:37.722963 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 24284
2026-09-28T04:47:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2844 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:47:37 dist-test-slave-j5c3 krb5kdc[24500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:47:37 dist-test-slave-j5c3 krb5kdc[24500](info): set up 2 sockets
Sep 28 04:47:37 dist-test-slave-j5c3 krb5kdc[24500](info): commencing operation
krb5kdc: starting...
W20260928 04:47:38.764703 1931 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/krb5kdc/test-user.keytab.
Sep 28 04:47:38 dist-test-slave-j5c3 krb5kdc[24500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:47:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:47:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.226.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
Entry for principal kudu/127.1.226.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab.
I20260928 04:47:38.903936 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:32889
--webserver_interface=127.1.226.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:37105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:32889,127.1.226.253:37109,127.1.226.252:38651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:47:38.988013 24513 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 04:47:38.988322 24513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:38.988356 24513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:38.988400 24513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:38.989631 24513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:38.989708 24513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:38.989730 24513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:38.989750 24513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:38.989778 24513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:38.991339 24513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:37105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:32889,127.1.226.253:37109,127.1.226.252:38651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.254.keytab
--principal=kudu/127.1.226.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.254:32889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.226.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:38.991636 24513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:38.991887 24513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:47:38.997218 24513 server_base.cc:1034] running on GCE node
W20260928 04:47:38.997179 24521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:38.997174 24519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:38.997184 24518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:38.997659 24513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:38.998045 24513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:38.999276 24513 hybrid_clock.cc:648] HybridClock initialized: now 1790570858999283 us; error 54 us; skew 500 ppm
Sep 28 04:47:39 dist-test-slave-j5c3 krb5kdc[24500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570859, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:39.006440 24513 init.cc:377] Logged in from keytab as kudu/127.1.226.254@KRBTEST.COM (short username kudu)
I20260928 04:47:39.007084 24513 webserver.cc:460] Webserver started at http://127.1.226.254:46019/ using document root <none> and password file <none>
I20260928 04:47:39.007359 24513 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:39.007436 24513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:39.007596 24513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:39.008957 24513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-0/data/instance:
uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be"
format_stamp: "Formatted at 2026-09-28 04:47:39 on dist-test-slave-j5c3"
server_key: "5d51b13936259a38bbeebe45a76807b1"
server_key_iv: "530eed66c8de548ad28d63c7879f7d47"
server_key_version: "encryptionkey@0"
I20260928 04:47:39.009382 24513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-0/wal/instance:
uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be"
format_stamp: "Formatted at 2026-09-28 04:47:39 on dist-test-slave-j5c3"
server_key: "5d51b13936259a38bbeebe45a76807b1"
server_key_iv: "530eed66c8de548ad28d63c7879f7d47"
server_key_version: "encryptionkey@0"
I20260928 04:47:39.010913 24513 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:47:39.011785 24528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:39.012109 24513 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:47:39.012291 24513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-0/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-0/wal
uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be"
format_stamp: "Formatted at 2026-09-28 04:47:39 on dist-test-slave-j5c3"
server_key: "5d51b13936259a38bbeebe45a76807b1"
server_key_iv: "530eed66c8de548ad28d63c7879f7d47"
server_key_version: "encryptionkey@0"
I20260928 04:47:39.012372 24513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:39.031997 24513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:39.034226 24513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:39.034350 24513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:39.038909 24513 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.254:32889
I20260928 04:47:39.038932 24580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.254:32889 every 8 connection(s)
I20260928 04:47:39.039382 24513 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-0/data/info.pb
I20260928 04:47:39.040530 24581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:39.042061 24581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.254" port: 32889 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 04:47:39 dist-test-slave-j5c3 krb5kdc[24500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570859, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:47:39.049242 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 24513
I20260928 04:47:39.049434 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-0/wal/instance
I20260928 04:47:39.049712 1931 external_mini_cluster.cc:1428] Setting key 777b9b131c0fb01291c4946f8d422d9b
I20260928 04:47:39.053692 24584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.042793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49164 (local address 127.1.226.254:32889)
0928 04:47:39.042955 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:39.042964 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:39.043143 (+ 179us) server_negotiation.cc:407] Connection header received
0928 04:47:39.043632 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:39.043634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:39.043652 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:39.043748 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:39.044294 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.045185 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.046092 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.046430 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.046526 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:39.049058 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:39.049068 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:39.049068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:39.049077 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:39.051571 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.052143 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.052146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.052147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.052310 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.052601 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.052601 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.052602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.052901 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:39.052975 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:39.053286 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 04:47:39.053508 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
I20260928 04:47:39.053702 24583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.042752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:32889 (local address 127.0.0.1:49164)
0928 04:47:39.043012 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:39.043045 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:39.043487 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:39.043762 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:39.043764 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:39.044026 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:39.044206 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:39.044210 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.045294 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.045295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:39.045991 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:39.045993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.046574 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.046575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:39.046636 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:39.047236 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:47:39.047244 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:39.048897 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:39.051693 (+ 2796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:39.051702 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:39.051705 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:39.052033 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:39.052390 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:39.052391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:39.052391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:39.052524 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:39.052981 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:39.052989 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:39.053076 (+ 87us) client_negotiation.cc:769] Sending connection context
0928 04:47:39.053184 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 04:47:39.053401 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":102,"threads_started":1}
I20260928 04:47:39.054665 24581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.253" port: 37109 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:47:39.055155 24583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.054999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:37109 (local address 127.0.0.1:56856)
0928 04:47:39.055057 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:39.055118 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.253:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 04:47:39.055366 24529 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.226.253:37109: connect: Connection refused (error 111)
I20260928 04:47:39.055943 24583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.055815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:37109 (local address 127.0.0.1:56872)
0928 04:47:39.055875 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:39.055909 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.253:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 04:47:39.056105 24581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.226.253:37109: Network error: Client connection negotiation failed: client connection to 127.1.226.253:37109: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.1.226.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.253.keytab.
Entry for principal kudu/127.1.226.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.253.keytab.
I20260928 04:47:39.079360 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.253:37109
--webserver_interface=127.1.226.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.253.keytab
--principal=kudu/127.1.226.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:37105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:32889,127.1.226.253:37109,127.1.226.252:38651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:47:39.105466 24581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.226.253" port: 37109 } attempt: 1
I20260928 04:47:39.106020 24583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.105852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:37109 (local address 127.0.0.1:56876)
0928 04:47:39.105928 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:39.105980 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.253:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 04:47:39.106423 24583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.106331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:37109 (local address 127.0.0.1:56880)
0928 04:47:39.106375 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:39.106404 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.253:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 04:47:39.106600 24581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.226.253:37109: Network error: Client connection negotiation failed: client connection to 127.1.226.253:37109: connect: Connection refused (error 111)
W20260928 04:47:39.163887 24588 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 04:47:39.164233 24588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:39.164255 24588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:39.164278 24588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:39.165524 24588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:39.165578 24588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:39.165592 24588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:39.165606 24588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:39.165618 24588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:39.167203 24588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:37105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:32889,127.1.226.253:37109,127.1.226.252:38651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.253.keytab
--principal=kudu/127.1.226.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.253:37109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.226.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:39.167486 24588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:39.167750 24588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:39.172977 24596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:39.173053 24594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:39.173053 24593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:39.173111 24588 server_base.cc:1034] running on GCE node
I20260928 04:47:39.173503 24588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:39.173805 24588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:39.174787 24581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.226.253" port: 37109 } attempt: 2
I20260928 04:47:39.174993 24588 hybrid_clock.cc:648] HybridClock initialized: now 1790570859174983 us; error 35 us; skew 500 ppm
I20260928 04:47:39.175222 24583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.175057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:37109 (local address 127.0.0.1:56890)
0928 04:47:39.175132 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:39.175182 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.253:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 04:47:39.175732 24583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.175612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:37109 (local address 127.0.0.1:56894)
0928 04:47:39.175666 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:39.175697 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.253:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 04:47:39.175843 24581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.226.253:37109: Network error: Client connection negotiation failed: client connection to 127.1.226.253:37109: connect: Connection refused (error 111)
Sep 28 04:47:39 dist-test-slave-j5c3 krb5kdc[24500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570859, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:39.177783 24588 init.cc:377] Logged in from keytab as kudu/127.1.226.253@KRBTEST.COM (short username kudu)
I20260928 04:47:39.178262 24588 webserver.cc:460] Webserver started at http://127.1.226.253:37157/ using document root <none> and password file <none>
I20260928 04:47:39.178496 24588 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:39.178565 24588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:39.178723 24588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:39.179806 24588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-1/data/instance:
uuid: "132145bbd0254190ad7befe099ccef95"
format_stamp: "Formatted at 2026-09-28 04:47:39 on dist-test-slave-j5c3"
server_key: "d23d4ac613129b07220d1f30cca66419"
server_key_iv: "bcaf66195a79c08c11c3776a5560f41b"
server_key_version: "encryptionkey@0"
I20260928 04:47:39.180262 24588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-1/wal/instance:
uuid: "132145bbd0254190ad7befe099ccef95"
format_stamp: "Formatted at 2026-09-28 04:47:39 on dist-test-slave-j5c3"
server_key: "d23d4ac613129b07220d1f30cca66419"
server_key_iv: "bcaf66195a79c08c11c3776a5560f41b"
server_key_version: "encryptionkey@0"
I20260928 04:47:39.181697 24588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:47:39.182575 24603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:39.182794 24588 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:47:39.182936 24588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-1/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-1/wal
uuid: "132145bbd0254190ad7befe099ccef95"
format_stamp: "Formatted at 2026-09-28 04:47:39 on dist-test-slave-j5c3"
server_key: "d23d4ac613129b07220d1f30cca66419"
server_key_iv: "bcaf66195a79c08c11c3776a5560f41b"
server_key_version: "encryptionkey@0"
I20260928 04:47:39.183019 24588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:39.207015 24588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:39.209172 24588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:39.209344 24588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:39.213614 24588 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.253:37109
I20260928 04:47:39.213656 24655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.253:37109 every 8 connection(s)
I20260928 04:47:39.214052 24588 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-1/data/info.pb
I20260928 04:47:39.214807 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 24588
I20260928 04:47:39.214989 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-1/wal/instance
I20260928 04:47:39.215118 24656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:39.215214 1931 external_mini_cluster.cc:1428] Setting key f81760ec3938b12d0827351ae68c4e33
I20260928 04:47:39.217125 24656 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.254" port: 32889 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.226.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.226.252@KRBTEST.COM" created.
Sep 28 04:47:39 dist-test-slave-j5c3 krb5kdc[24500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570859, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.253@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:47:39.230819 24584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.217901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49180 (local address 127.1.226.254:32889)
0928 04:47:39.217969 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:39.217971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:39.218169 (+ 198us) server_negotiation.cc:407] Connection header received
0928 04:47:39.218696 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:39.218698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:39.218713 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:39.218760 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:39.219450 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.220295 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.222072 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.222500 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.222580 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:39.227362 (+ 4782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:39.227363 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:39.227364 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:39.227373 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:39.229206 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.229758 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.229760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.229760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.229820 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.230135 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.230136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.230137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.230269 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:39.230347 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:39.230638 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 04:47:39.230723 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 04:47:39.231021 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.217864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:32889 (local address 127.0.0.1:49180)
0928 04:47:39.218027 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:39.218080 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:39.218582 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:39.218899 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:39.218903 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:39.219161 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:39.219352 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:39.219356 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.220400 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.220401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:39.221965 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:39.221968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.222629 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.222631 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:39.222849 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:39.223466 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:47:39.223477 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:39.227211 (+ 3734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:39.229307 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:39.229315 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:39.229318 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:39.229657 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:39.229913 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:39.229914 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:39.229915 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:39.230053 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:39.230339 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:39.230346 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:39.230501 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 04:47:39.230607 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 04:47:39.230776 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20260928 04:47:39.231621 24656 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.253" port: 37109 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 04:47:39 dist-test-slave-j5c3 krb5kdc[24500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570859, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.253@KRBTEST.COM for kudu/127.1.226.253@KRBTEST.COM
I20260928 04:47:39.240631 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.231926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:37109 (local address 127.0.0.1:56906)
0928 04:47:39.232141 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:39.232152 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:39.232215 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:39.232347 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:39.232348 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:39.232525 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:39.232572 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:39.232574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.233421 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.233422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:39.233927 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:39.233928 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.234423 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.234424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:39.234488 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:39.234857 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:47:39.234861 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:39.237283 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:39.239398 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:39.239399 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:39.239399 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:39.239561 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:39.239794 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:39.239795 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:39.239796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:39.239853 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:39.240272 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:39.240272 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:39.240316 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 04:47:39.240405 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 04:47:39.240517 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200}
I20260928 04:47:39.240748 24661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.231987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56906 (local address 127.1.226.253:37109)
0928 04:47:39.232195 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:39.232200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:39.232222 (+ 22us) server_negotiation.cc:407] Connection header received
0928 04:47:39.232267 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:39.232270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:39.232284 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:39.232349 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:39.232655 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.233330 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.234016 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.234331 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.234425 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:39.237399 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:39.237409 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:39.237410 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:39.237417 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:39.239297 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.239659 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.239663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.239664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.239727 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.239925 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.239925 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.239926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.240140 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:39.240297 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:39.240438 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 04:47:39.240609 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":135,"thread_start_us":106,"threads_started":1}
I20260928 04:47:39.241457 24656 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.252" port: 38651 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:47:39.241811 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.241710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:38651 (local address 127.0.0.1:36366)
0928 04:47:39.241745 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:39.241790 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260928 04:47:39.241981 24606 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.226.252:38651: connect: Connection refused (error 111)
I20260928 04:47:39.242486 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.242415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:38651 (local address 127.0.0.1:36370)
0928 04:47:39.242441 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:39.242466 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260928 04:47:39.242606 24656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.226.252:38651: Network error: Client connection negotiation failed: client connection to 127.1.226.252:38651: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.226.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.252.keytab.
Entry for principal kudu/127.1.226.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.252.keytab.
I20260928 04:47:39.244027 1931 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky6vTKM/build/release/bin/kudu
/tmp/dist-test-tasky6vTKM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.252:38651
--webserver_interface=127.1.226.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.252.keytab
--principal=kudu/127.1.226.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.226.212:37105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.226.254:32889,127.1.226.253:37109,127.1.226.252:38651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:47:39.267006 24581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.226.253" port: 37109 } attempt: 3
Sep 28 04:47:39 dist-test-slave-j5c3 krb5kdc[24500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570859, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for kudu/127.1.226.253@KRBTEST.COM
I20260928 04:47:39.274814 24583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.267432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:37109 (local address 127.0.0.1:56918)
0928 04:47:39.267499 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:39.267512 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:39.267559 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:39.267722 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:39.267722 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:39.267878 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:39.267946 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:39.267948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.268896 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.268897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:39.269432 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:39.269434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.269915 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.269915 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:39.269974 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:39.270447 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:47:39.270451 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:39.271906 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:39.273813 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:39.273814 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:39.273814 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:39.273981 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:39.274218 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:39.274219 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:39.274219 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:39.274282 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:39.274545 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:39.274546 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:39.274586 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 04:47:39.274639 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 04:47:39.274741 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 04:47:39.274871 24661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.267517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56918 (local address 127.1.226.253:37109)
0928 04:47:39.267571 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:39.267573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:39.267585 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:47:39.267617 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:39.267618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:39.267630 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:39.267710 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:39.268023 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.268796 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.269511 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.269841 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.269912 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:39.271993 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:39.271995 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:39.271995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:39.272003 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:39.273724 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.274056 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.274057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.274058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.274120 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.274345 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.274346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.274346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.274480 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:39.274548 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:39.274685 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 04:47:39.274780 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 04:47:39.275508 24581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.252" port: 38651 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:47:39.275857 24583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.275753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:38651 (local address 127.0.0.1:36376)
0928 04:47:39.275804 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:39.275832 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 04:47:39.276252 24583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.276126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:38651 (local address 127.0.0.1:36384)
0928 04:47:39.276184 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:39.276227 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 04:47:39.276382 24581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.226.252:38651: Network error: Client connection negotiation failed: client connection to 127.1.226.252:38651: connect: Connection refused (error 111)
I20260928 04:47:39.291869 24656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.226.252" port: 38651 } attempt: 1
I20260928 04:47:39.292390 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.292256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:38651 (local address 127.0.0.1:36396)
0928 04:47:39.292323 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:39.292362 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 04:47:39.292742 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.292641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:38651 (local address 127.0.0.1:36412)
0928 04:47:39.292690 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:39.292720 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 04:47:39.292842 24656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.226.252:38651: Network error: Client connection negotiation failed: client connection to 127.1.226.252:38651: connect: Connection refused (error 111)
I20260928 04:47:39.307544 24581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.226.252" port: 38651 } attempt: 1
I20260928 04:47:39.308065 24583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.307912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:38651 (local address 127.0.0.1:36420)
0928 04:47:39.307978 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:39.308035 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 04:47:39.308545 24583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.308406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:38651 (local address 127.0.0.1:36426)
0928 04:47:39.308487 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:39.308522 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 04:47:39.308687 24581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.226.252:38651: Network error: Client connection negotiation failed: client connection to 127.1.226.252:38651: connect: Connection refused (error 111)
W20260928 04:47:39.330232 24663 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 04:47:39.330458 24663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:47:39.330477 24663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:47:39.330498 24663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:47:39.331712 24663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:47:39.331758 24663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:47:39.331771 24663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:47:39.331784 24663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:47:39.331796 24663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:47:39.333395 24663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.226.212:37105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.226.254:32889,127.1.226.253:37109,127.1.226.252:38651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/krb5kdc/kudu_127.1.226.252.keytab
--principal=kudu/127.1.226.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.226.252:38651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.226.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:47:39.333663 24663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:47:39.333930 24663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:47:39.339061 24669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:39.339080 24668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:47:39.339084 24671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:47:39.339874 24663 server_base.cc:1034] running on GCE node
I20260928 04:47:39.340106 24663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:47:39.340451 24663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:47:39.341639 24663 hybrid_clock.cc:648] HybridClock initialized: now 1790570859341594 us; error 78 us; skew 500 ppm
Sep 28 04:47:39 dist-test-slave-j5c3 krb5kdc[24500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570859, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:47:39.344592 24663 init.cc:377] Logged in from keytab as kudu/127.1.226.252@KRBTEST.COM (short username kudu)
I20260928 04:47:39.345091 24663 webserver.cc:460] Webserver started at http://127.1.226.252:42265/ using document root <none> and password file <none>
I20260928 04:47:39.345337 24663 fs_manager.cc:359] Metadata directory not provided
I20260928 04:47:39.345407 24663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:47:39.345577 24663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:47:39.346745 24663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-2/data/instance:
uuid: "3c0583183ef742099f80bac5b9ab4871"
format_stamp: "Formatted at 2026-09-28 04:47:39 on dist-test-slave-j5c3"
server_key: "c12a5351c28ca9ee83e9540121530e7a"
server_key_iv: "95d4c29e16ba9be5e7e89e65185f9d38"
server_key_version: "encryptionkey@0"
I20260928 04:47:39.347110 24663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-2/wal/instance:
uuid: "3c0583183ef742099f80bac5b9ab4871"
format_stamp: "Formatted at 2026-09-28 04:47:39 on dist-test-slave-j5c3"
server_key: "c12a5351c28ca9ee83e9540121530e7a"
server_key_iv: "95d4c29e16ba9be5e7e89e65185f9d38"
server_key_version: "encryptionkey@0"
I20260928 04:47:39.348564 24663 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:47:39.349483 24678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:47:39.349689 24663 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:47:39.349846 24663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-2/data,/tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-2/wal
uuid: "3c0583183ef742099f80bac5b9ab4871"
format_stamp: "Formatted at 2026-09-28 04:47:39 on dist-test-slave-j5c3"
server_key: "c12a5351c28ca9ee83e9540121530e7a"
server_key_iv: "95d4c29e16ba9be5e7e89e65185f9d38"
server_key_version: "encryptionkey@0"
I20260928 04:47:39.349931 24663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:47:39.360999 24656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.226.252" port: 38651 } attempt: 2
I20260928 04:47:39.361460 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.361302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:38651 (local address 127.0.0.1:36440)
0928 04:47:39.361373 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:39.361418 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 04:47:39.361810 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.361731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:38651 (local address 127.0.0.1:36444)
0928 04:47:39.361761 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:39.361789 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.226.252:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 04:47:39.361930 24656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.226.252:38651: Network error: Client connection negotiation failed: client connection to 127.1.226.252:38651: connect: Connection refused (error 111)
I20260928 04:47:39.362511 24663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:47:39.364670 24663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:47:39.364791 24663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:47:39.368729 24663 rpc_server.cc:307] RPC server started. Bound to: 127.1.226.252:38651
I20260928 04:47:39.368848 24730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.226.252:38651 every 8 connection(s)
I20260928 04:47:39.369158 24663 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-2/data/info.pb
I20260928 04:47:39.370157 24731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:47:39.371829 24731 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.254" port: 32889 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 04:47:39 dist-test-slave-j5c3 krb5kdc[24500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570859, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.252@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:47:39.379212 1931 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky6vTKM/build/release/bin/kudu as pid 24663
I20260928 04:47:39.379364 1931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky6vTKM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570585693787-1931-0/minicluster-data/master-2/wal/instance
I20260928 04:47:39.379608 1931 external_mini_cluster.cc:1428] Setting key eb00797be8a683c4a9c37e2b0b792450
I20260928 04:47:39.382269 24584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.372672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49194 (local address 127.1.226.254:32889)
0928 04:47:39.372771 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:39.372774 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:39.372968 (+ 194us) server_negotiation.cc:407] Connection header received
0928 04:47:39.373488 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:39.373489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:39.373500 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:39.373568 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:39.374215 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.374700 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.375760 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.375992 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.376065 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:39.378654 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:39.378656 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:39.378657 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:39.378677 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:39.380548 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.381115 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.381116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.381116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.381177 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.381516 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.381517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.381518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.381644 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:39.381716 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:39.382071 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 04:47:39.382166 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 04:47:39.382507 24733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.372615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.254:32889 (local address 127.0.0.1:49194)
0928 04:47:39.372823 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:39.372869 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:39.373367 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:39.373626 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:39.373629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:39.373895 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:39.374110 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:39.374114 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.374798 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.374799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:39.375642 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:39.375644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.376124 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.376125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:39.376354 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:39.376953 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:47:39.376970 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:39.378529 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:39.380632 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:39.380639 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:39.380641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:39.381026 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:39.381249 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:39.381250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:39.381251 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:39.381413 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:39.381728 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:39.381736 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:39.381916 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 04:47:39.382044 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 04:47:39.382229 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":124,"threads_started":1}
I20260928 04:47:39.383361 24731 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.253" port: 37109 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:47:39.383826 24581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.226.252" port: 38651 } attempt: 2
Sep 28 04:47:39 dist-test-slave-j5c3 krb5kdc[24500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.252@KRBTEST.COM
Sep 28 04:47:39 dist-test-slave-j5c3 krb5kdc[24500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570859, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.252@KRBTEST.COM for kudu/127.1.226.253@KRBTEST.COM
Sep 28 04:47:39 dist-test-slave-j5c3 krb5kdc[24500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570859, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.254@KRBTEST.COM for kudu/127.1.226.252@KRBTEST.COM
I20260928 04:47:39.391840 24735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.380144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36458 (local address 127.1.226.252:38651)
0928 04:47:39.380727 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:39.380733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:39.380770 (+ 37us) server_negotiation.cc:407] Connection header received
0928 04:47:39.380812 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:39.380815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:39.380832 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:39.380910 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:39.381781 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.382706 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.383854 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.384357 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.384444 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:39.387242 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:39.387534 (+ 292us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:39.387535 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:39.387544 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:39.389500 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.389938 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.389941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.389942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.390009 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.390289 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.390290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.390291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.391321 (+ 1030us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:39.391517 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:39.391569 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 04:47:39.391699 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":94,"threads_started":1}
I20260928 04:47:39.392406 24661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.383775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56920 (local address 127.1.226.253:37109)
0928 04:47:39.383853 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:39.383856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:39.383883 (+ 27us) server_negotiation.cc:407] Connection header received
0928 04:47:39.383947 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:39.383948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:39.383958 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:39.384018 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:39.384369 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.385049 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.385974 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.386204 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.386284 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:39.389207 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:39.389208 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:39.389208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:39.389216 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:39.391049 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.391548 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.391550 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.391550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.391610 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.391829 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.391830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.391831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.391979 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:39.392019 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:39.392235 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 04:47:39.392310 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 04:47:39.392406 24733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.383686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.253:37109 (local address 127.0.0.1:56920)
0928 04:47:39.383835 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:39.383847 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:39.383896 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:39.384023 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:39.384024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:39.384243 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:39.384307 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:39.384308 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.385141 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.385142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:39.385914 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:39.385916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.386286 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.386287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:39.386356 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:39.387120 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:47:39.387124 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:39.389094 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:39.391135 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:39.391136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:39.391136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:39.391451 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:39.391687 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:39.391688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:39.391688 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:39.391749 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:39.392064 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:39.392065 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:39.392104 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 04:47:39.392206 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 04:47:39.392310 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260928 04:47:39.392783 24731 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.226.252" port: 38651 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:47:39.393680 24583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.384073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:38651 (local address 127.0.0.1:36460)
0928 04:47:39.384185 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:39.384194 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:39.384244 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:39.384497 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:39.384498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:39.384641 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:39.384703 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:39.384707 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.385712 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.385713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:39.386553 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:39.386557 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.387067 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.387068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:39.387141 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:39.387534 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:47:39.387538 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:39.389094 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:39.391961 (+ 2867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:39.391962 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:39.391963 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:39.392231 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:39.392962 (+ 731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:39.392963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:39.392963 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:39.393036 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:39.393356 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:39.393357 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:39.393412 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 04:47:39.393474 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 04:47:39.393583 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 04:47:39.393730 24736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.384185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36460 (local address 127.1.226.252:38651)
0928 04:47:39.384352 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:39.384354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:39.384362 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:47:39.384381 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:39.384381 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:39.384392 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:39.384783 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:39.384829 (+ 46us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.385620 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.386646 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.386986 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.387053 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:39.389182 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:39.389183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:39.389184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:39.389191 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:39.391852 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.392316 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.392317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.392317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.392388 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.393114 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.393115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.393115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.393260 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:39.393329 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:39.393510 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 04:47:39.393616 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":111,"threads_started":1}
I20260928 04:47:39.395469 24581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb310118d0c84ef7a03d3ffc0b5eb2be: Bootstrap starting.
I20260928 04:47:39.396566 24581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb310118d0c84ef7a03d3ffc0b5eb2be: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:39.396883 24581 log.cc:824] T 00000000000000000000000000000000 P bb310118d0c84ef7a03d3ffc0b5eb2be: Log is configured to *not* fsync() on all Append() calls
Sep 28 04:47:39 dist-test-slave-j5c3 krb5kdc[24500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570859, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.252@KRBTEST.COM for kudu/127.1.226.252@KRBTEST.COM
I20260928 04:47:39.397828 24581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb310118d0c84ef7a03d3ffc0b5eb2be: No bootstrap required, opened a new log
Sep 28 04:47:39 dist-test-slave-j5c3 krb5kdc[24500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.254@KRBTEST.COM
I20260928 04:47:39.400055 24581 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb310118d0c84ef7a03d3ffc0b5eb2be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 32889 } } peers { permanent_uuid: "132145bbd0254190ad7befe099ccef95" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 37109 } } peers { permanent_uuid: "3c0583183ef742099f80bac5b9ab4871" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 38651 } }
I20260928 04:47:39.400243 24581 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb310118d0c84ef7a03d3ffc0b5eb2be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb310118d0c84ef7a03d3ffc0b5eb2be, State: Initialized, Role: FOLLOWER
I20260928 04:47:39.400354 24581 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb310118d0c84ef7a03d3ffc0b5eb2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 32889 } } peers { permanent_uuid: "132145bbd0254190ad7befe099ccef95" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 37109 } } peers { permanent_uuid: "3c0583183ef742099f80bac5b9ab4871" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 38651 } }
I20260928 04:47:39.400697 24738 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb310118d0c84ef7a03d3ffc0b5eb2be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 32889 } } peers { permanent_uuid: "132145bbd0254190ad7befe099ccef95" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 37109 } } peers { permanent_uuid: "3c0583183ef742099f80bac5b9ab4871" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 38651 } } }
I20260928 04:47:39.400772 24581 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb310118d0c84ef7a03d3ffc0b5eb2be [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:39.400887 24738 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb310118d0c84ef7a03d3ffc0b5eb2be [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:47:39.401142 24735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.393095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36476 (local address 127.1.226.252:38651)
0928 04:47:39.393127 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:39.393128 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:39.393320 (+ 192us) server_negotiation.cc:407] Connection header received
0928 04:47:39.393340 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:39.393340 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:39.393351 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:39.393411 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:39.393720 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.394448 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.395105 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.395384 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.395448 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:39.397906 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:39.397908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:39.397908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:39.397916 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:39.399567 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.400237 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.400238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.400239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.400298 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.400489 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.400489 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.400490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.400684 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:39.400749 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:39.400944 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 04:47:39.401026 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 04:47:39.401443 24733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.393059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:38651 (local address 127.0.0.1:36476)
0928 04:47:39.393210 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:39.393220 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:39.393278 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:39.393413 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:39.393414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:39.393595 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:39.393636 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:39.393638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.394525 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.394526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:39.395004 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:39.395005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.395466 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.395467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:39.395528 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:39.396421 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:47:39.396426 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:39.397810 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:39.399907 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:39.399908 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:39.399908 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:39.400102 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:39.400369 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:39.400370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:39.400370 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:39.400426 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:39.400795 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:39.400796 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:39.400837 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 04:47:39.401237 (+ 400us) client_negotiation.cc:240] Negotiation successful
0928 04:47:39.401333 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260928 04:47:39.402285 24584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.392730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.1.226.254:32889)
0928 04:47:39.392796 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:39.392799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:39.392808 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:47:39.393114 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:39.393115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:39.393127 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:39.393157 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:39.394347 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.394988 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.395941 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.396285 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.396353 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:39.399163 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:39.399164 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:39.399165 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:39.399173 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:39.400822 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.401411 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.401413 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.401413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.401482 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.401719 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.401721 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.401721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.401863 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:39.402044 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:39.402093 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 04:47:39.402173 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 04:47:39.402956 24731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c0583183ef742099f80bac5b9ab4871: Bootstrap starting.
I20260928 04:47:39.403529 24731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c0583183ef742099f80bac5b9ab4871: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:39.403735 24731 log.cc:824] T 00000000000000000000000000000000 P 3c0583183ef742099f80bac5b9ab4871: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:39.404822 24731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c0583183ef742099f80bac5b9ab4871: No bootstrap required, opened a new log
I20260928 04:47:39.406728 24731 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c0583183ef742099f80bac5b9ab4871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 32889 } } peers { permanent_uuid: "132145bbd0254190ad7befe099ccef95" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 37109 } } peers { permanent_uuid: "3c0583183ef742099f80bac5b9ab4871" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 38651 } }
I20260928 04:47:39.406879 24731 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c0583183ef742099f80bac5b9ab4871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0583183ef742099f80bac5b9ab4871, State: Initialized, Role: FOLLOWER
I20260928 04:47:39.406955 24731 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c0583183ef742099f80bac5b9ab4871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 32889 } } peers { permanent_uuid: "132145bbd0254190ad7befe099ccef95" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 37109 } } peers { permanent_uuid: "3c0583183ef742099f80bac5b9ab4871" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 38651 } }
I20260928 04:47:39.407248 24731 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c0583183ef742099f80bac5b9ab4871 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:39.407260 24751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c0583183ef742099f80bac5b9ab4871 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 32889 } } peers { permanent_uuid: "132145bbd0254190ad7befe099ccef95" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 37109 } } peers { permanent_uuid: "3c0583183ef742099f80bac5b9ab4871" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 38651 } } }
I20260928 04:47:39.407405 24751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c0583183ef742099f80bac5b9ab4871 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:47:39.405579 24513 master_runner.cc:186] Error getting master registration for 127.1.226.253:37109: OK, instance_id { permanent_uuid: "132145bbd0254190ad7befe099ccef95" instance_seqno: 1790570859212451 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 04:47:39.408546 24513 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 04:47:39.409655 24762 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c0583183ef742099f80bac5b9ab4871: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:47:39.409827 24762 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 28 04:47:39 dist-test-slave-j5c3 krb5kdc[24500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.226.253@KRBTEST.COM
W20260928 04:47:39.410614 24749 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bb310118d0c84ef7a03d3ffc0b5eb2be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:47:39.410671 24749 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:47:39.414318 24661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.404825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56926 (local address 127.1.226.253:37109)
0928 04:47:39.404869 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:39.404871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:39.404913 (+ 42us) server_negotiation.cc:407] Connection header received
0928 04:47:39.404991 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:39.404991 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:39.405003 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:39.405055 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:39.405725 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.406447 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.407432 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.407776 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.407852 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:39.410910 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:39.410912 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:39.410912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:39.410921 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:39.412673 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.413074 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.413075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.413075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.413133 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.413350 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.413351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.413351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.413487 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:39.413663 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:39.414130 (+ 467us) server_negotiation.cc:299] Negotiation successful
0928 04:47:39.414209 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:47:39.412038 24663 master_runner.cc:186] Error getting master registration for 127.1.226.253:37109: OK, instance_id { permanent_uuid: "132145bbd0254190ad7befe099ccef95" instance_seqno: 1790570859212451 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 04:47:39.414388 24663 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 04:47:39.415836 1931 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:47:39.417639 24555 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49210
I20260928 04:47:39.417798 24555 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb310118d0c84ef7a03d3ffc0b5eb2be [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 04:47:39.417855 24555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb310118d0c84ef7a03d3ffc0b5eb2be [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:39.418867 24555 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb310118d0c84ef7a03d3ffc0b5eb2be [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 32889 } } peers { permanent_uuid: "132145bbd0254190ad7befe099ccef95" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 37109 } } peers { permanent_uuid: "3c0583183ef742099f80bac5b9ab4871" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 38651 } }
I20260928 04:47:39.419152 24555 leader_election.cc:288] T 00000000000000000000000000000000 P bb310118d0c84ef7a03d3ffc0b5eb2be [CANDIDATE]: Term 1 election: Requested vote from peers 132145bbd0254190ad7befe099ccef95 (127.1.226.253:37109), 3c0583183ef742099f80bac5b9ab4871 (127.1.226.252:38651)
I20260928 04:47:39.419486 24705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3c0583183ef742099f80bac5b9ab4871"
I20260928 04:47:39.419627 24705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c0583183ef742099f80bac5b9ab4871 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:39.420835 24705 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c0583183ef742099f80bac5b9ab4871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb310118d0c84ef7a03d3ffc0b5eb2be in term 1.
I20260928 04:47:39.421128 24531 leader_election.cc:302] T 00000000000000000000000000000000 P bb310118d0c84ef7a03d3ffc0b5eb2be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c0583183ef742099f80bac5b9ab4871, bb310118d0c84ef7a03d3ffc0b5eb2be; no voters:
I20260928 04:47:39.421319 24738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb310118d0c84ef7a03d3ffc0b5eb2be [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:47:39.421489 24738 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb310118d0c84ef7a03d3ffc0b5eb2be [term 1 LEADER]: Becoming Leader. State: Replica: bb310118d0c84ef7a03d3ffc0b5eb2be, State: Running, Role: LEADER
I20260928 04:47:39.421597 24738 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb310118d0c84ef7a03d3ffc0b5eb2be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 32889 } } peers { permanent_uuid: "132145bbd0254190ad7befe099ccef95" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 37109 } } peers { permanent_uuid: "3c0583183ef742099f80bac5b9ab4871" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 38651 } }
I20260928 04:47:39.422062 24738 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb310118d0c84ef7a03d3ffc0b5eb2be [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb310118d0c84ef7a03d3ffc0b5eb2be. Latest consensus state: current_term: 1 leader_uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 32889 } } peers { permanent_uuid: "132145bbd0254190ad7befe099ccef95" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 37109 } } peers { permanent_uuid: "3c0583183ef742099f80bac5b9ab4871" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 38651 } } }
I20260928 04:47:39.422158 24738 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb310118d0c84ef7a03d3ffc0b5eb2be [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:39.422465 24773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:47:39.422690 24773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:47:39.424636 24705 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c0583183ef742099f80bac5b9ab4871 [term 1 FOLLOWER]: Refusing update from remote peer bb310118d0c84ef7a03d3ffc0b5eb2be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:47:39.425010 24738 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bb310118d0c84ef7a03d3ffc0b5eb2be [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0583183ef742099f80bac5b9ab4871" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 38651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:47:39.426453 24751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c0583183ef742099f80bac5b9ab4871 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb310118d0c84ef7a03d3ffc0b5eb2be. Latest consensus state: current_term: 1 leader_uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 32889 } } peers { permanent_uuid: "132145bbd0254190ad7befe099ccef95" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 37109 } } peers { permanent_uuid: "3c0583183ef742099f80bac5b9ab4871" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 38651 } } }
I20260928 04:47:39.426555 24751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c0583183ef742099f80bac5b9ab4871 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:47:39.427577 24738 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb310118d0c84ef7a03d3ffc0b5eb2be [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 32889 } } peers { permanent_uuid: "132145bbd0254190ad7befe099ccef95" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 37109 } } peers { permanent_uuid: "3c0583183ef742099f80bac5b9ab4871" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 38651 } } }
I20260928 04:47:39.427665 24738 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb310118d0c84ef7a03d3ffc0b5eb2be [sys.catalog]: This master's current role is: LEADER
I20260928 04:47:39.428076 24751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c0583183ef742099f80bac5b9ab4871 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 32889 } } peers { permanent_uuid: "132145bbd0254190ad7befe099ccef95" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 37109 } } peers { permanent_uuid: "3c0583183ef742099f80bac5b9ab4871" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 38651 } } }
I20260928 04:47:39.428261 24773 catalog_manager.cc:1347] Generated new cluster ID: 5a35b8bd545849ae8296580155f9e32e
I20260928 04:47:39.428319 24773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:47:39.432322 24751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c0583183ef742099f80bac5b9ab4871 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:47:39.436256 24776 mvcc.cc:205] Tried to move back new op lower bound from 7334178240200196096 to 7334178240191823872. Current Snapshot: MvccSnapshot[applied={T|T < 7334178240200196096}]
I20260928 04:47:39.442530 24735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.421432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36492 (local address 127.1.226.252:38651)
0928 04:47:39.422311 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:39.422313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:39.432351 (+ 10038us) server_negotiation.cc:407] Connection header received
0928 04:47:39.432494 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:39.432495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:39.432510 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:39.432566 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:39.433409 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.434213 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.435966 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.436287 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.436326 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:39.438901 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:39.438902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:39.438903 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:39.438911 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:39.440639 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.441323 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.441338 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.441338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.441399 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.441734 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.441734 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.441735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.441865 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:39.441939 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:39.442324 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 04:47:39.442432 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":856}
I20260928 04:47:39.443295 24773 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:47:39.444617 24773 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:47:39.449863 24584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.422358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49220 (local address 127.1.226.254:32889)
0928 04:47:39.422415 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:39.422417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:39.432571 (+ 10154us) server_negotiation.cc:407] Connection header received
0928 04:47:39.433481 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:39.433482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:39.433493 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:39.433523 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:39.434231 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.434965 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.435935 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.436236 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.436278 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:39.445172 (+ 8894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:39.445173 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:39.445174 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:39.445182 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:39.447872 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.448346 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.448348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.448348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.448412 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.448885 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.448888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.448888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.449026 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:39.449134 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:39.449620 (+ 486us) server_negotiation.cc:299] Negotiation successful
0928 04:47:39.449729 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 04:47:39.453037 24656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.226.252" port: 38651 } attempt: 3
I20260928 04:47:39.453258 24661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.422202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56930 (local address 127.1.226.253:37109)
0928 04:47:39.422283 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:39.422286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:39.436733 (+ 14447us) server_negotiation.cc:407] Connection header received
0928 04:47:39.436883 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:39.436884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:39.436896 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:39.436945 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:39.445745 (+ 8800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.446495 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.447531 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.447896 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.447959 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:39.450335 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:39.450336 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:39.450337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:39.450347 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:39.452000 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.452425 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.452427 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.452427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.452486 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.452709 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.452710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.452710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.452853 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:39.452943 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:39.453070 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 04:47:39.453147 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 04:47:39.455034 24773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb310118d0c84ef7a03d3ffc0b5eb2be: Generated new TSK 0
I20260928 04:47:39.455437 24773 catalog_manager.cc:1514] Initializing in-progress tserver states...
Sep 28 04:47:39 dist-test-slave-j5c3 krb5kdc[24500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570859, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.226.253@KRBTEST.COM for kudu/127.1.226.252@KRBTEST.COM
I20260928 04:47:39.465297 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.453352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.226.252:38651 (local address 127.0.0.1:36502)
0928 04:47:39.453476 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 04:47:39.453487 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:47:39.453536 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:47:39.454019 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:47:39.454020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:47:39.454182 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:47:39.454232 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:39.454234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.455171 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.455172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:39.455803 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:47:39.455805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.456318 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.456319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:47:39.456395 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:39.456914 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:47:39.456920 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:47:39.458433 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:47:39.464064 (+ 5631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:39.464066 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:39.464066 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:39.464371 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:39.464593 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:47:39.464594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:47:39.464594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:47:39.464655 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:47:39.464945 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:47:39.464946 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:47:39.465002 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 04:47:39.465072 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 04:47:39.465180 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 04:47:39.465737 24735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:39.453848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36502 (local address 127.1.226.252:38651)
0928 04:47:39.453904 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:39.453907 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:39.453916 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:47:39.453940 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:39.453941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:39.453952 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:39.454009 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:39.454315 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.455081 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.455902 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:39.456237 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:39.456304 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:39.458533 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:47:39.458535 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:47:39.458535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:47:39.458543 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:47:39.461313 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.464459 (+ 3146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.464460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.464461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.464526 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:47:39.464720 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:47:39.464721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:47:39.464721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:47:39.464854 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:47:39.464916 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:39.465569 (+ 653us) server_negotiation.cc:299] Negotiation successful
0928 04:47:39.465646 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:47:39.466993 24656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 132145bbd0254190ad7befe099ccef95: Bootstrap starting.
I20260928 04:47:39.467631 24656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 132145bbd0254190ad7befe099ccef95: Neither blocks nor log segments found. Creating new log.
I20260928 04:47:39.467917 24656 log.cc:824] T 00000000000000000000000000000000 P 132145bbd0254190ad7befe099ccef95: Log is configured to *not* fsync() on all Append() calls
I20260928 04:47:39.468684 24656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 132145bbd0254190ad7befe099ccef95: No bootstrap required, opened a new log
I20260928 04:47:39.470140 24656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 132145bbd0254190ad7befe099ccef95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 32889 } } peers { permanent_uuid: "132145bbd0254190ad7befe099ccef95" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 37109 } } peers { permanent_uuid: "3c0583183ef742099f80bac5b9ab4871" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 38651 } }
I20260928 04:47:39.470254 24656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 132145bbd0254190ad7befe099ccef95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 132145bbd0254190ad7befe099ccef95, State: Initialized, Role: FOLLOWER
I20260928 04:47:39.470309 24656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 132145bbd0254190ad7befe099ccef95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 32889 } } peers { permanent_uuid: "132145bbd0254190ad7befe099ccef95" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 37109 } } peers { permanent_uuid: "3c0583183ef742099f80bac5b9ab4871" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 38651 } }
I20260928 04:47:39.470624 24656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 132145bbd0254190ad7befe099ccef95 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:47:39.470770 24629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 132145bbd0254190ad7befe099ccef95 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:47:39.471720 24629 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 132145bbd0254190ad7befe099ccef95 [term 1 FOLLOWER]: Refusing update from remote peer bb310118d0c84ef7a03d3ffc0b5eb2be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:47:39.471966 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "132145bbd0254190ad7befe099ccef95"
I20260928 04:47:39.472779 24630 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 132145bbd0254190ad7befe099ccef95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb310118d0c84ef7a03d3ffc0b5eb2be in term 1.
I20260928 04:47:39.472909 24783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 132145bbd0254190ad7befe099ccef95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 32889 } } peers { permanent_uuid: "132145bbd0254190ad7befe099ccef95" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 37109 } } peers { permanent_uuid: "3c0583183ef742099f80bac5b9ab4871" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 38651 } } }
I20260928 04:47:39.472992 24783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 132145bbd0254190ad7befe099ccef95 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:47:39.473133 24769 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bb310118d0c84ef7a03d3ffc0b5eb2be [LEADER]: Connected to new peer: Peer: permanent_uuid: "132145bbd0254190ad7befe099ccef95" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 37109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:47:39.476994 24783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 132145bbd0254190ad7befe099ccef95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb310118d0c84ef7a03d3ffc0b5eb2be. Latest consensus state: current_term: 1 leader_uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 32889 } } peers { permanent_uuid: "132145bbd0254190ad7befe099ccef95" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 37109 } } peers { permanent_uuid: "3c0583183ef742099f80bac5b9ab4871" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 38651 } } }
I20260928 04:47:39.477087 24783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 132145bbd0254190ad7befe099ccef95 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:47:39.477120 24783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 132145bbd0254190ad7befe099ccef95 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 32889 } } peers { permanent_uuid: "132145bbd0254190ad7befe099ccef95" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 37109 } } peers { permanent_uuid: "3c0583183ef742099f80bac5b9ab4871" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 38651 } } }
I20260928 04:47:39.477166 24783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 132145bbd0254190ad7befe099ccef95 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 04:47:39.481012 24794 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 132145bbd0254190ad7befe099ccef95: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:47:39.481144 24794 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:47:39.481789 24796 mvcc.cc:205] Tried to move back new op lower bound from 7334178240323641344 to 7334178240191823872. Current Snapshot: MvccSnapshot[applied={T|T < 7334178240200196096}]
W20260928 04:47:39.497208 24735 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:47:39.491509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36506 (local address 127.1.226.252:38651)
0928 04:47:39.491559 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:39.491561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:39.496418 (+ 4857us) server_negotiation.cc:407] Connection header received
0928 04:47:39.496541 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:39.496542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:39.496555 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:39.496586 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:39.497133 (+ 547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36506: BlockingRecv error: recv got EOF from 127.0.0.1:36506 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 04:47:39.498023 24584 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:47:39.491938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49236 (local address 127.1.226.254:32889)
0928 04:47:39.491980 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:39.491982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:39.497477 (+ 5495us) server_negotiation.cc:407] Connection header received
0928 04:47:39.497591 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:39.497592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:39.497604 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:39.497635 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:39.497959 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49236: BlockingRecv error: recv got EOF from 127.0.0.1:49236 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 04:47:39.498795 24661 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:47:39.492335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56936 (local address 127.1.226.253:37109)
0928 04:47:39.492388 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:39.492391 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:39.498256 (+ 5865us) server_negotiation.cc:407] Connection header received
0928 04:47:39.498317 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:39.498318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:39.498331 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:39.498420 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:39.498727 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56936: BlockingRecv error: recv got EOF from 127.0.0.1:56936 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 04:47:39.501106 24661 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:47:39.500269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56950 (local address 127.1.226.253:37109)
0928 04:47:39.500316 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:39.500318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:39.500539 (+ 221us) server_negotiation.cc:407] Connection header received
0928 04:47:39.500686 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:39.500687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:39.500716 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:39.500746 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:39.501060 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56950: BlockingRecv error: recv got EOF from 127.0.0.1:56950 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 04:47:39.505319 24661 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:47:39.504526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56962 (local address 127.1.226.253:37109)
0928 04:47:39.504566 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:39.504568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:39.504843 (+ 275us) server_negotiation.cc:407] Connection header received
0928 04:47:39.504937 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:39.504938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:39.504948 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:39.504977 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:39.505273 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56962: BlockingRecv error: recv got EOF from 127.0.0.1:56962 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 04:47:39.509315 24661 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:47:39.508552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56976 (local address 127.1.226.253:37109)
0928 04:47:39.508588 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:39.508590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:39.508852 (+ 262us) server_negotiation.cc:407] Connection header received
0928 04:47:39.508944 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:39.508944 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:39.508955 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:39.508982 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:39.509271 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56976: BlockingRecv error: recv got EOF from 127.0.0.1:56976 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 04:47:39.515439 24661 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:47:39.514503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56982 (local address 127.1.226.253:37109)
0928 04:47:39.514543 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:39.514545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:39.514832 (+ 287us) server_negotiation.cc:407] Connection header received
0928 04:47:39.514933 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:39.514934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:39.514945 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:39.514991 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:39.515389 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56982: BlockingRecv error: recv got EOF from 127.0.0.1:56982 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 04:47:39.525775 24661 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:47:39.524834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56994 (local address 127.1.226.253:37109)
0928 04:47:39.524880 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:39.524882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:39.525146 (+ 264us) server_negotiation.cc:407] Connection header received
0928 04:47:39.525256 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:39.525257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:39.525271 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:39.525303 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:39.525715 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56994: BlockingRecv error: recv got EOF from 127.0.0.1:56994 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 04:47:39.544271 24661 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:47:39.543258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56998 (local address 127.1.226.253:37109)
0928 04:47:39.543305 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:39.543307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:39.543593 (+ 286us) server_negotiation.cc:407] Connection header received
0928 04:47:39.543711 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:39.543712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:39.543725 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:39.543759 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:39.544209 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56998: BlockingRecv error: recv got EOF from 127.0.0.1:56998 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 04:47:39.578761 24661 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:47:39.577792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57006 (local address 127.1.226.253:37109)
0928 04:47:39.577844 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:39.577847 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:39.578128 (+ 281us) server_negotiation.cc:407] Connection header received
0928 04:47:39.578231 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:39.578233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:39.578246 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:39.578277 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:39.578703 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57006: BlockingRecv error: recv got EOF from 127.0.0.1:57006 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 04:47:39.645340 24661 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:47:39.644386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57010 (local address 127.1.226.253:37109)
0928 04:47:39.644467 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:39.644470 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:39.644594 (+ 124us) server_negotiation.cc:407] Connection header received
0928 04:47:39.644635 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:39.644636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:39.644651 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:39.644737 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:39.645277 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57010: BlockingRecv error: recv got EOF from 127.0.0.1:57010 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 04:47:39.775420 24661 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:47:39.774616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57020 (local address 127.1.226.253:37109)
0928 04:47:39.774705 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:39.774707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:39.774719 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:47:39.774771 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:39.774772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:39.774786 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:39.774863 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:39.775314 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57020: BlockingRecv error: recv got EOF from 127.0.0.1:57020 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 04:47:39.900490 24769 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb310118d0c84ef7a03d3ffc0b5eb2be [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb310118d0c84ef7a03d3ffc0b5eb2be" member_type: VOTER last_known_addr { host: "127.1.226.254" port: 32889 } } peers { permanent_uuid: "132145bbd0254190ad7befe099ccef95" member_type: VOTER last_known_addr { host: "127.1.226.253" port: 37109 } } peers { permanent_uuid: "3c0583183ef742099f80bac5b9ab4871" member_type: VOTER last_known_addr { host: "127.1.226.252" port: 38651 } } }
I20260928 04:47:39.900657 24769 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb310118d0c84ef7a03d3ffc0b5eb2be [sys.catalog]: This master's current role is: LEADER
W20260928 04:47:40.033511 24661 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:47:40.032810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57030 (local address 127.1.226.253:37109)
0928 04:47:40.032892 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:40.032894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:40.032904 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:47:40.032936 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:40.032942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:40.032953 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:40.033022 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:47:40.033442 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57030: BlockingRecv error: recv got EOF from 127.0.0.1:57030 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 04:47:40.410312 24762 catalog_manager.cc:1259] Loaded cluster ID: 5a35b8bd545849ae8296580155f9e32e
I20260928 04:47:40.410395 24762 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3c0583183ef742099f80bac5b9ab4871: loading cluster ID for follower catalog manager: success
I20260928 04:47:40.411399 24762 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3c0583183ef742099f80bac5b9ab4871: acquiring CA information for follower catalog manager: success
I20260928 04:47:40.411731 24762 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3c0583183ef742099f80bac5b9ab4871: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 04:47:40.481691 24794 catalog_manager.cc:1259] Loaded cluster ID: 5a35b8bd545849ae8296580155f9e32e
I20260928 04:47:40.481818 24794 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 132145bbd0254190ad7befe099ccef95: loading cluster ID for follower catalog manager: success
I20260928 04:47:40.482801 24794 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 132145bbd0254190ad7befe099ccef95: acquiring CA information for follower catalog manager: success
I20260928 04:47:40.483150 24794 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 132145bbd0254190ad7befe099ccef95: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 04:47:40.549593 24866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:40.546808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57042 (local address 127.1.226.253:37109)
0928 04:47:40.546944 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:40.546946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:40.547070 (+ 124us) server_negotiation.cc:407] Connection header received
0928 04:47:40.547107 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:40.547108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:40.547163 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:40.547258 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:47:40.547408 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:40.547993 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:40.548803 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:40.549053 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:40.549116 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:40.549265 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:47:40.549391 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:47:40.549467 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:40.549500 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 04:47:40.549521 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260928 04:47:40.554520 24872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:40.551743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36514 (local address 127.1.226.252:38651)
0928 04:47:40.551845 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:40.551847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:40.551857 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:47:40.551980 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:40.551981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:40.552042 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:40.552132 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:47:40.552296 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:40.552879 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:40.553681 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:40.553907 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:40.553985 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:40.554155 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:47:40.554275 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:47:40.554367 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:40.554392 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 04:47:40.554417 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":50,"threads_started":1}
I20260928 04:47:40.559513 24872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:40.556349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.1.226.252:38651)
0928 04:47:40.556390 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:40.556392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:40.556691 (+ 299us) server_negotiation.cc:407] Connection header received
0928 04:47:40.556750 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:40.556751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:40.556762 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:40.556812 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:47:40.556994 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:40.557467 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:40.558573 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:40.558889 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:40.558973 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:40.559153 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:47:40.559200 (+ 47us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:47:40.559278 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:40.559387 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 04:47:40.559416 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:47:40.559805 24866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:40.556298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57058 (local address 127.1.226.253:37109)
0928 04:47:40.556439 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:40.556442 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:40.556653 (+ 211us) server_negotiation.cc:407] Connection header received
0928 04:47:40.556756 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:40.556758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:40.556772 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:40.556812 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:47:40.556967 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:40.557694 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:40.558697 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:40.559060 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:40.559150 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:40.559499 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:47:40.559537 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:47:40.559569 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:40.559703 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 04:47:40.559733 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260928 04:47:40.560909 24878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:47:40.556320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49252 (local address 127.1.226.254:32889)
0928 04:47:40.556450 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 04:47:40.556452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:47:40.556584 (+ 132us) server_negotiation.cc:407] Connection header received
0928 04:47:40.556668 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:47:40.556669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:47:40.556730 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:47:40.556813 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:47:40.556906 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:40.557701 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:40.559569 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:47:40.560207 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:47:40.560277 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:47:40.560494 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:47:40.560583 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:47:40.560633 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 04:47:40.560783 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 04:47:40.560803 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
I20260928 04:47:40.562995 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 24513
I20260928 04:47:40.568789 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 24588
I20260928 04:47:40.573907 1931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky6vTKM/build/release/bin/kudu with pid 24663
2026-09-28T04:47:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2852 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5697 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (274886 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST